Method for hospital visit guidance for medical treatment for active thyroid eye disease, and system for performing same

ABSTRACT

According to the present application, a computer-implemented method of predicting thyroid eye disease is disclosed. The method comprising: preparing a conjunctival hyperemia prediction model, a conjunctival edema prediction model, a lacrimal edema prediction model, an eyelid redness prediction model, and an eyelid edema prediction model, obtaining a facial image of an object, obtaining a first processed image and a second processed image from the facial image, wherein the first processed image is different from the second processed image, obtaining predicted values for each of a conjunctival hyperemia, a conjunctival edema and a lacrimal edema by applying the first processed image to the conjunctival hyperemia prediction model, the conjunctival edema prediction model, and the lacrimal edema prediction model, and obtaining predicted values for each of an eyelid redness and an eyelid edema by applying the second processed image to the eyelid redness prediction model and the eyelid edema prediction model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/939,040 filed on Sep. 7, 2022, which is a continuation ofInternational Application No. PCT/KR2022/009356 filed on Jun. 29, 2022,which claims priority to Korean Patent Application No. 10-2021-0085542filed on Jun. 30, 2021 and Korean Patent Application No. 10-2022-0079770filed on Jun. 29, 2022, the entire contents of which are hereinincorporated by reference.

TECHNICAL FIELD

The present disclosure relates to a method for hospital visit guidancefor medical treatment for active thyroid eye disease, and a system forperforming the method.

BACKGROUND ART

An eye disease is a disease that occurs in the eyeball and surroundingparts. Many people in the world have been affected with eye diseases,and in severe cases, eye diseases cause great inconvenience in life,such as damage to eyesight, so it is necessary to monitor the occurrenceor extent of the eye diseases.

In the meantime, an eye disease may be one of several complicationscaused by other diseases. For example, thyroid eye disease is acomplication caused by thyroid dysfunction.

When thyroid eye disease becomes worse, the eyeball protrudes and cannotbe treated without surgery. Therefore, early diagnosis of thyroid eyedisease is very important for the treatment of thyroid eye disease.However, it is difficult to diagnose thyroid eye disease early becausethe disease does not show clear prognostic symptoms. In the medicalcommunity, efforts have been made to diagnose thyroid eye disease earlythrough an evaluation method with the clinical activity score (CAS),which was proposed in 1989.

In determining the clinical activity score for thyroid eye disease, atotal of seven items are considered, and the seven items are 1)spontaneous retrobulbar pain, 2) pain on an attempted upward or downwardgaze, 3) redness of an eyelid, 4) redness of a conjunctiva, 5) swellingof an eyelid, 6) swelling of a conjunctiva, and 7) swelling of alacrimal caruncle.

In order to determine a clinical activity score, it is essential that anindividual visits a hospital or clinic in person and the doctor performsa medical examination through interview and observation with the nakedeye. For example, spontaneous retrobulbar pain and pain on an attemptedupward or downward gaze can be checked through interview by a doctor,and redness of an eyelid, redness of a conjunctiva, swelling of aneyelid, swelling of a conjunctiva, and swelling of a lacrimal carunclecan be checked by a doctor's observation with the naked eye. Thedoctor's medical examination with the naked eye and interview method fordetermining a clinical activity score require a hospital visit by apatient in person for a diagnosis of thyroid eye disease, as aprecondition, so it is difficult to diagnose thyroid eye disease early.

Accordingly, it is desired to develop a method of enabling individualsto recognize a risk of eye disease more easily and quickly without ahospital visit in person so that continuous monitoring can be performed,and of informing a patient of a risk of eye disease to induce thepatient to visit the hospital when necessary.

DISCLOSURE Technical Problem

The disclosure in the present application is directed to providing alearning model used in predicting a clinical activity score for thyroideye disease by using an image that is obtained with a digital camerathat ordinary people can use rather than a professional medicaldiagnostic device.

In addition, the disclosure in the present application is directed toproviding a method and a system for enabling ordinary people tocontinuously monitor a clinical activity score for thyroid eye diseasewithout a doctor's help and a hospital visit in person.

In addition, the disclosure in the present application is directed toproviding a method of recommending a hospital visit for medicaltreatment for active thyroid eye disease according to a result ofmonitoring a clinical activity score, and a system for performing themethod.

Technical problems to be solved by the present application are notlimited to the aforementioned technical problems and other technicalproblems which are not mentioned will be clearly understood by thoseskilled in the art from the present specification and the accompanyingdrawings.

Technical Solution

According to one aspect of the present application, acomputer-implemented method of predicting thyroid eye disease isdisclosed. The method comprising: preparing a conjunctival hyperemiaprediction model, a conjunctival edema prediction model, a lacrimaledema prediction model, an eyelid redness prediction model, and aneyelid edema prediction model, obtaining a facial image of an object,obtaining a first processed image and a second processed image from thefacial image, wherein the first processed image is different from thesecond processed image, obtaining predicted values for each of aconjunctival hyperemia, a conjunctival edema and a lacrimal edema byapplying the first processed image to the conjunctival hyperemiaprediction model, the conjunctival edema prediction model, and thelacrimal edema prediction model, obtaining predicted values for each ofan eyelid redness and an eyelid edema by applying the second processedimage to the eyelid redness prediction model and the eyelid edemaprediction model, and determining a possibility of the object havingthyroid eye disease based on the predicted values for the conjunctivalhyperemia, the conjunctival edema, the lacrimal edema, the eyelidredness, and the eyelid edema. Wherein the first processed image is animage masked a region corresponding an inner of an outline of an irisand a region corresponding an outer of an outline of an eye and croppedalong a first region comprising the outline of the eye based on positioninformation of pixels corresponding the outline of the iris comprised inthe eye and position information of pixels corresponding the outline ofthe eye, and wherein the second processed image is an image croppedalong a second region that is wider than the first region based onposition information of pixels corresponding the outline of the iriscomprised in the eye and position information of pixels correspondingthe outline of the eye.

In some embodiments, wherein the position information of pixelscorresponding the outline of the iris comprised in the eye and theposition information of pixels corresponding the outline of the eye areobtained by segmentation model.

In some embodiments, wherein the first processed image comprises a firstprocessed left eye image and a first processed right eye image, andwherein the second processed image comprises a second processed left eyeimage and a second processed right eye image. [15] In some embodiments,wherein the conjunctival hyperemia prediction model comprises a left eyeconjunctival hyperemia prediction model and a right eye conjunctivalhyperemia prediction model, wherein the conjunctival edema predictionmodel comprises a left eye conjunctival edema prediction model and aright eye conjunctival edema prediction model, wherein the lacrimaledema prediction model comprises a left eye lacrimal edema predictionmodel and a right eye lacrimal edema prediction model, wherein theeyelid redness prediction model comprises a left eyelid rednessprediction model and a right eye eyelid redness prediction model, andwherein the eyelid edema prediction model comprises a left eye eyelidedema prediction model and a right eye eyelid edema prediction model.

In some embodiments, wherein the predicted value for the conjunctivalhyperemia is determined based on a result obtained by inputting thefirst processed left eye image to the left eye conjunctival hyperemiaprediction model and a result obtained by inputting the first processedright eye image to the right eye conjunctival hyperemia predictionmodel, wherein the predicted value for the conjunctival edema isdetermined based on a result obtained by inputting the first processedleft eye image to the left eye conjunctival edema prediction model and aresult obtained by inputting the first processed right eye image to theright eye conjunctival edema prediction model, wherein the predictedvalue for the lacrimal edema is determined based on a result obtained byinputting the first processed left eye image to the left eye lacrimaledema prediction model and a result obtained by inputting the firstprocessed right eye image to the right eye lacrimal edema predictionmodel, wherein the predicted value for the eyelid redness is determinedbased on a result obtained by inputting the second processed left eyeimage to the left eye eyelid redness prediction model and a resultobtained by inputting the second processed right eye image to the righteye eyelid redness prediction model, and wherein the predicted value forthe eyelid edema is determined based on a result obtained by inputtingthe second processed left eye image to the left eye eyelid edemaprediction model and a result obtained by inputting the second processedright eye image to the right eye eyelid edema prediction model.

In some embodiments, the method further comprising: processing any oneof the first processed left eye image and the first processed right eyeimage by inverting left and right, and processing any one of the secondprocessed left eye image and the second processed right eye image byinverting left and right.

In some embodiments, wherein the predicted value for the conjunctivalhyperemia is determined based on results obtained by inputting the imagelaterally inverted and by inputting the image not laterally inverted tothe conjunctival hyperemia prediction model, wherein the predicted valuefor the conjunctival edema is determined based on results obtained byinputting the image laterally inverted and by inputting the image notlaterally inverted to the conjunctival edema prediction model, whereinthe predicted value for the lacrimal edema is determined based onresults obtained by inputting the image laterally inverted and byinputting the image not laterally inverted to the lacrimal edemaprediction model, wherein the predicted value for the eyelid redness isdetermined based on results obtained by inputting the image laterallyinverted and by inputting the image not laterally inverted to the eyelidredness prediction model, and wherein the predicted value for the eyelidedema is determined based on results obtained by inputting the imagelaterally inverted and by inputting the image not laterally inverted tothe eyelid edema prediction model.

In some embodiments, the method further comprising: resizing the firstprocessed left eye image and the first processed right eye image, andresizing the second processed left eye image and the second processedright eye image.

Advantageous Effects

According to the disclosure in the present application, a clinicalactivity score for thyroid eye disease can be predicted using imagesobtained through a digital camera that ordinary people can use, ratherthan a professional medical diagnostic device.

In addition, according to the disclosure in the present application,ordinary people can continuously monitor a clinical activity score forthyroid eye disease without a doctor's help and a hospital visit inperson, and a hospital visit is recommended when necessary.

DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a system for predicting a clinicalactivity score for thyroid eye disease according to an embodimentdescribed in the present application.

FIG. 2 is a block diagram illustrating a user terminal provided in thepresent application.

FIG. 3 is a block diagram illustrating a server described in the presentapplication.

FIG. 4 is a diagram illustrating an eye and the surrounding tissues thatare exposed to the outside so that the eye and the surrounding tissuesare captured by a camera when a picture of the face is taken using thecamera.

FIG. 5 is a diagram illustrating an eyeball exposed to the outside.

FIG. 6 is a diagram illustrating an outline of an eye.

FIG. 7 is a diagram illustrating a cornea exposed to the outside.

FIG. 8 is a diagram illustrating a conjunctiva exposed to the outside.

FIG. 9A is a diagram illustrating a facial image.

FIG. 9B is a diagram illustrating a two-eye image.

FIG. 10A is a diagram illustrating a two-eye image.

FIG. 10B is a diagram illustrating a left eye image and a right eyeimage.

FIG. 11 is a diagram illustrating X_(max), X_(min), Y_(max), and Y_(min)of outline pixels.

FIG. 12 is a diagram illustrating a second cropped region determined.

FIG. 13A is a diagram illustrating an example of a second cropped image.

FIG. 13B is a diagram illustrating an example of a second cropped image.

FIGS. 14A and 14B are diagrams illustrating examples of third croppedimages.

FIGS. 15 and 16 are diagrams illustrating iris segmentation.

FIG. 17 is a diagram illustrating eye outline segmentation.

FIG. 18A is a diagram illustrating an example of a first masking image.

FIG. 18B is a diagram illustrating an example of a first masking image.

FIG. 19 is a diagram illustrating an example of a second masking image.

FIGS. 20 to 22 are diagrams illustrating various examples of originalimages and laterally inverted images.

FIG. 23 is a flowchart illustrating a conjunctival hyperemia predictionmethod.

FIG. 24 is a flowchart illustrating a conjunctival edema predictionmethod.

FIG. 25 is a flowchart illustrating a lacrimal edema prediction method.

FIG. 26 is a flowchart illustrating an eyelid redness prediction method.

FIG. 27 is a flowchart illustrating an eyelid edema prediction method.

FIG. 28 is a diagram illustrating a method of predicting a clinicalactivity score for thyroid eye disease.

FIG. 29 is a diagram illustrating a method of continuously monitoring aclinical activity score for thyroid eye disease, and a method ofrecommending a hospital visit on the basis of the former.

DETAILED DESCRIPTION

The above-described objectives, features, and advantages of the presentapplication will be more apparent from the following detaileddescription with reference to the accompanying drawings. In addition,various modifications may be made to the present application, andvarious embodiments of the present application may be practiced.Therefore, specific embodiments will be described in detail below withreference to the accompanying drawings.

Throughout the specification, the same reference numerals denote thesame elements in principle. In addition, elements having the samefunction within the same scope illustrated in the drawings of theembodiments are described using the same reference numerals, and aredundant description will be omitted.

A detailed description of a well-known function or configurationrelating to the present application is omitted when determined asobfuscating the nature and gist of the present application. In addition,throughout the present specification, the terms first, second, and so onare used only to distinguish from one element to another.

In addition, the terms “module” and “part” that are used to name anelement in the description below are used considering only the ease withwhich the present specification is written. The terms are not intendedas having different special meanings or functions and thus may be usedindividually or interchangeably.

In the following embodiments, an expression used in the singularencompasses the expression of the plural, unless it has a clearlydifferent meaning in the context.

In the following embodiments, it is to be understood that terms such as“including”, “having”, etc. are intended to indicate the existence offeatures or elements disclosed in the specification, and are notintended to preclude the possibility that one or more other features orelements may be added.

Sizes of elements in the drawings may be exaggerated or reduced forconvenience of description. For example, any size and thickness of eachelement shown in the drawings are shown for convenience of description,and the present disclosure is not limited thereto.

In a case in which a particular embodiment is realized otherwise, aparticular process may be performed out of the order described. Forexample, two processes described in succession may be performedsubstantially simultaneously, or may proceed in the order opposite tothe order described.

In the following embodiments, when elements are referred to as beingconnected to each other, the elements are directly connected to eachother or the elements are indirectly connected to each other withintervening elements therebetween. For example, in the presentspecification, when elements are referred to as being electricallyconnected to each other, the elements are directly electricallyconnected to each other or the elements are indirectly electricallyconnected with intervening elements therebetween.

According to one aspect of the present application, acomputer-implemented method of predicting thyroid eye disease isdisclosed. The method comprising: preparing a conjunctival hyperemiaprediction model, a conjunctival edema prediction model, a lacrimaledema prediction model, an eyelid redness prediction model, and aneyelid edema prediction model, obtaining a facial image of an object,obtaining a first processed image and a second processed image from thefacial image, wherein the first processed image is different from thesecond processed image, obtaining predicted values for each of aconjunctival hyperemia, a conjunctival edema and a lacrimal edema byapplying the first processed image to the conjunctival hyperemiaprediction model, the conjunctival edema prediction model, and thelacrimal edema prediction model, obtaining predicted values for each ofan eyelid redness and an eyelid edema by applying the second processedimage to the eyelid redness prediction model and the eyelid edemaprediction model, and determining a possibility of the object havingthyroid eye disease based on the predicted values for the conjunctivalhyperemia, the conjunctival edema, the lacrimal edema, the eyelidredness, and the eyelid edema. Wherein the first processed image is animage masked a region corresponding an inner of an outline of an irisand a region corresponding an outer of an outline of an eye and croppedalong a first region comprising the outline of the eye based on positioninformation of pixels corresponding the outline of the iris comprised inthe eye and position information of pixels corresponding the outline ofthe eye, and wherein the second processed image is an image croppedalong a second region that is wider than the first region based onposition information of pixels corresponding the outline of the iriscomprised in the eye and position information of pixels correspondingthe outline of the eye.

In some embodiments, wherein the position information of pixelscorresponding the outline of the iris comprised in the eye and theposition information of pixels corresponding the outline of the eye areobtained by segmentation model.

In some embodiments, wherein the first processed image comprises a firstprocessed left eye image and a first processed right eye image, andwherein the second processed image comprises a second processed left eyeimage and a second processed right eye image.

In some embodiments, wherein the conjunctival hyperemia prediction modelcomprises a left eye conjunctival hyperemia prediction model and a righteye conjunctival hyperemia prediction model, wherein the conjunctivaledema prediction model comprises a left eye conjunctival edemaprediction model and a right eye conjunctival edema prediction model,wherein the lacrimal edema prediction model comprises a left eyelacrimal edema prediction model and a right eye lacrimal edemaprediction model, wherein the eyelid redness prediction model comprisesa left eyelid redness prediction model and a right eye eyelid rednessprediction model, and wherein the eyelid edema prediction modelcomprises a left eye eyelid edema prediction model and a right eyeeyelid edema prediction model.

In some embodiments, wherein the predicted value for the conjunctivalhyperemia is determined based on a result obtained by inputting thefirst processed left eye image to the left eye conjunctival hyperemiaprediction model and a result obtained by inputting the first processedright eye image to the right eye conjunctival hyperemia predictionmodel, wherein the predicted value for the conjunctival edema isdetermined based on a result obtained by inputting the first processedleft eye image to the left eye conjunctival edema prediction model and aresult obtained by inputting the first processed right eye image to theright eye conjunctival edema prediction model, wherein the predictedvalue for the lacrimal edema is determined based on a result obtained byinputting the first processed left eye image to the left eye lacrimaledema prediction model and a result obtained by inputting the firstprocessed right eye image to the right eye lacrimal edema predictionmodel, wherein the predicted value for the eyelid redness is determinedbased on a result obtained by inputting the second processed left eyeimage to the left eye eyelid redness prediction model and a resultobtained by inputting the second processed right eye image to the righteye eyelid redness prediction model, and wherein the predicted value forthe eyelid edema is determined based on a result obtained by inputtingthe second processed left eye image to the left eye eyelid edemaprediction model and a result obtained by inputting the second processedright eye image to the right eye eyelid edema prediction model.

In some embodiments, the method further comprising: processing any oneof the first processed left eye image and the first processed right eyeimage by inverting left and right, and processing any one of the secondprocessed left eye image and the second processed right eye image byinverting left and right.

In some embodiments, wherein the predicted value for the conjunctivalhyperemia is determined based on results obtained by inputting the imagelaterally inverted and by inputting the image not laterally inverted tothe conjunctival hyperemia prediction model, wherein the predicted valuefor the conjunctival edema is determined based on results obtained byinputting the image laterally inverted and by inputting the image notlaterally inverted to the conjunctival edema prediction model, whereinthe predicted value for the lacrimal edema is determined based onresults obtained by inputting the image laterally inverted and byinputting the image not laterally inverted to the lacrimal edemaprediction model, wherein the predicted value for the eyelid redness isdetermined based on results obtained by inputting the image laterallyinverted and by inputting the image not laterally inverted to the eyelidredness prediction model, and wherein the predicted value for the eyelidedema is determined based on results obtained by inputting the imagelaterally inverted and by inputting the image not laterally inverted tothe eyelid edema prediction model.

In some embodiments, the method further comprising: resizing the firstprocessed left eye image and the first processed right eye image, andresizing the second processed left eye image and the second processedright eye image.

According to the present application, disclosed is a system forpredicting a clinical activity score (CAS) for a user's thyroid eyedisease, and for providing, on the basis of the CAS, guidance about thenecessity for the user to visit the hospital.

1. Whole System

(1) Hardware Construction of System

FIG. 1 is a diagram illustrating a system for predicting a clinicalactivity score for thyroid eye disease according to an embodimentdescribed in the present application.

Referring to FIG. 1 , the system 1 includes a plurality of userterminals 10 and a server 20.

Hereinafter, the plurality of user terminals 10 and the server 20 willbe described in detail.

(2) Functions of User Terminals

The plurality of user terminals 10 transmit information to the server 20over various networks, and receive information from the server 20.

The plurality of user terminals 10 obtain images of users' uppereyelids, lower eyelids, and eyeballs exposed to the outside by the uppereyelids and the lower eyelids (hereinafter, referred to as eye images).The plurality of user terminals 10 may perform necessary processing onthe obtained eye images, or may transmit the obtained eye images or theprocessed eye images to the server 20.

The plurality of user terminals 10 may receive, from the server 20,prediction results about clinical activity scores processed by theserver 20.

(3) Functions of Server

The server 20 transmits information to the plurality of user terminals10 over various networks, and receive information from the plurality ofuser terminals 10.

The server 20 may receive the eye images from the plurality of userterminals 10. Herein, the server 20 may process the eye images.Alternatively, the server 20 may receive the processed eye images.

The server 20 may obtain, on the basis of the processed eye images,prediction results about clinical activity scores for users' thyroid eyediseases.

The server 20 may transmit the prediction results about the clinicalactivity scores to the plurality of user terminals 10.

(4) Software Construction of System

In order for the system 1 to operate, several software constructions arerequired.

To perform communication between the user terminals 10 and the server20, terminal software needs to be installed on the plurality of userterminals 10, and server software needs to be installed on the server20.

In order to perform pre-processing necessary for the eye images, variouspreprocessing algorithms may be used.

A plurality of learning models for predicting clinical activity scoreson the basis of the preprocessed eye images may be used.

The plurality of preprocessing algorithms may be run by the terminalsoftware installed on the user terminals 10, or may be run by thesoftware installed on the server 20. Alternatively, some of theplurality of preprocessing algorithms may be executed by the userterminals 10, and the others may be executed by the server 20.

The plurality of learning models may be run by the software installed onthe server 20.

Alternatively, the plurality of learning models may be run by theterminal software installed on the user terminals 10. Alternatively,some of the plurality of learning models may be executed by the userterminals 10, and the others may be executed by the server 20.

(5) Elements of User Terminal

FIG. 2 is a block diagram illustrating a user terminal described in thepresent application.

Referring to FIG. 2 , a user terminal 10 described in the presentapplication includes an output part 110, a communication part 120, amemory 130, a camera 140, and a controller 150.

The output part 110 outputs various types of information according tocontrol commands of the controller 150. According to an embodiment, theoutput part 110 may include a display 112 for outputting informationvisually to a user. Alternatively, although not shown in the drawings, aspeaker for outputting information audibly to a user, and a vibrationmotor for outputting information tactually to a user may be included.

The communication part 120 may include a wireless communication moduleand/or a wired communication module. Herein, examples of the wirelesscommunication module may include a Wi-Fi communication module, acellular communication module, etc.

The memory 130 stores therein executable code readable by the controller150, processed result values, necessary data, etc. Examples of thememory 130 may include a hard disk drive (HDD), a solid state disk(SSD), a silicon disk drive (SDD), ROM, RAM, etc. The memory 130 maystore therein the above-described terminal software, and may storetherein executable codes for realizing the above-described variouspreprocessing algorithms and/or learning models. Furthermore, the memory130 may store therein an eye image obtained through the camera 140, thepreprocessed eye images, etc.

The camera 140 is a digital camera, and may include an image sensor andan image processor. The image sensor is a device for converting anoptical image into electrical signals, and may be provided as a chip inwhich multiple photodiodes are integrated. Examples of the image sensormay include a charge-coupled device (CCD), a complementarymetal-oxide-semiconductor (CMOS), etc. In the meantime, the imageprocessor may perform image processing on captured results, and maygenerate image information.

The controller 150 may include at least one processor. Herein, each ofthe processors may perform a predetermined operation by executing atleast one instruction stored in the memory 130. Specifically, thecontroller 150 may process information according to the terminalsoftware, the preprocessing algorithms, and/or the learning modelsrunning on the user terminal 10. In the meantime, the controller 150controls the overall operation of the user terminal 10.

Although not shown in the drawings, the user terminal 10 may include auser input part. The user terminal 10 may receive, from a user, varioustypes of information required for the operation of the user terminal 10through the user input part.

(6) Elements of Server

FIG. 3 is a block diagram illustrating a server described in the presentapplication.

Referring to FIG. 3 , the server 20 described in the present applicationincludes a communication part 210, a memory 220, and a controller 230.

The communication part 210 may include a wireless communication moduleand/or a wired communication module. Herein, examples of the wirelesscommunication module may include a Wi-Fi communication module, acellular communication module, etc.

The memory 220 stores therein executable code readable by the controller230, processed result values, necessary data, etc. Examples of thememory 220 may include a hard disk drive (HDD), a solid state disk(SSD), a silicon disk drive (SDD), ROM, RAM, etc. The memory 220 maystore therein the above-described server software, and may store thereinexecutable codes for realizing the above-described various preprocessingalgorithms and/or learning models. Furthermore, the memory 220 may storetherein an eye image received from the user terminal 10, thepreprocessed eye images, etc.

The controller 230 may include at least one processor. Herein, each ofthe processors may perform a predetermined operation by executing atleast one instruction stored in the memory 220.

Specifically, the controller 230 may process information according tothe server software, the preprocessing algorithms, and/or the learningmodels running on the server 20. In the meantime, the controller 230controls the overall operation of the server 20.

Hereinafter, in order to more clearly and easily understand thetechnology described in the present application, an eye, an eyeball, andthe tissues near the eyeball including an upper eyelid, a lower eyelid,and a lacrimal caruncle will be briefly described, and the terms relatedto an eye and the surroundings used in the present specification will bedefined.

2. Construction of Eye and Definition of Terms

(1) Eyeball and Surrounding Tissues

FIG. 4 is a diagram illustrating an eye and the surrounding tissues thatare exposed to the outside so that the eye and the surrounding tissuesare captured by a camera when a picture of the face is taken using thecamera.

FIG. 4 shows eyelids (the upper eyelid and the lower eyelid), a lacrimalcaruncle, and a conjunctiva and a cornea partially exposed and partiallycovered by the upper eyelid, the lower eyelid, and the lacrimalcaruncle.

In general, an eye or eyeball is larger than that shown in FIG. 4 .However, an eyeball is protected from the outside by tissues, such asthe upper eyelid, and the lower eyelid, and thus, only part of theeyeball is exposed to the outside even when the person has his or hereye open.

(2) Definition of Terms

Conjunctiva, White of Eye

Hereinafter, a conjunctiva generally corresponds to the position of thewhite of an eye, so the terms the conjunctiva and the white of the eyemay be used interchangeably.

Cornea, Iris

Hereinafter, a cornea generally corresponds to the position of the irisof an eye, so the terms cornea and iris may be used interchangeably. Inthe meantime, in the present specification, the term ‘iris’ is used in asense including a pupil region.

Eyelids

Eyelids are two, upper and lower folds of skin covering the front partof an eyeball. Eyelids are also called palpebrae. The eyelid above aneyeball is called the upper eyelid, and the eyelid below the eyeball iscalled the lower eyelid. The outer surface is skin and the inner surfaceis a conjunctiva, and therebetween, there are muscles moving theeyelids, and tarsal plates containing meibomian glands, which aresebaceous glands, thus maintaining the shape of the eyelids. The eyelidsprotect the eyeball, and simultaneously, make the eyeball clean withtears by blinking the eye or make the cornea shiny and transparent.

Eyebrow

An eyebrow refers to hairs grown in an arc along the bony ridge above aneye.

Eyelashes

Eyelashes refer to hairs about 10 mm in length on the edge of upper andlower eyelids.

Eyeball Exposed to Outside

Hereinafter, “the eyeball exposed to the outside” means the part notcovered by the upper eyelid, the lower eyelid, and the lacrimalcaruncle, that is, the part exposed to the outside by the upper eyelid,the lower eyelid, and the lacrimal caruncle when a person has his or hereye open. For example, the inside of the dotted line shown in FIG. 5 iscalled “the eyeball exposed to the outside”.

Outline of Eye

Hereinafter, “the outline of the eye” means the outline of the partincluding both the eyeball exposed to the outside and the lacrimalcaruncle region when a person has his or her eye open. That is, theoutline of the region that is a combination of the eyeball exposed tothe outside and the lacrimal caruncle is called “the outline of theeye”. For example, the dotted line shown in FIG. 6 is called “theoutline of the eye”.

Cornea Exposed to Outside (Iris Exposed to Outside)

Hereinafter, “the cornea exposed to the outside” means the cornea partnot covered by the upper eyelid and the lower eyelid, that is, thecornea part exposed to the outside by the upper eyelid and the lowereyelid, when a person has his or her eye open. For example, the insideof the dotted line shown in FIG. 7 is called “the cornea exposed to theoutside”.

Conjunctiva Exposed to Outside (White of Eye Exposed to Outside)

Hereinafter, “the conjunctiva exposed to the outside” means theconjunctiva part not covered by the upper eyelid, the lower eyelid, andthe lacrimal caruncle, that is, the conjunctiva part exposed to theoutside by the upper eyelid, the lower eyelid, and the lacrimalcaruncle, when a person has his or her eye open. For example, the insideof the dotted line shown in FIG. 8 is called “the conjunctiva exposed tothe outside”.

Hereinafter, various image preprocessing algorithms for performing imagepreprocessing described in the present application will be described.

3. Image Preprocessing Algorithms

(1) Necessity of Image Preprocessing

The present application is directed to providing a learning model forpredicting a clinical activity score for thyroid eye disease by using animage obtained by a digital camera that ordinary people can use, ratherthan a professional medical diagnostic device.

To this end, in predicting a clinical activity score for thyroid eyedisease, images that can be easily obtained by ordinary people foreyeballs and the tissues near the eyeballs need to be used. For example,an image analysis uses a digital image obtained by a digital camera or acamera built in a smartphone that can be easily used by ordinary peoplerather than a digital image obtained by a specialized medical deviceused in a medical institution.

Under this environment, a digital image obtained by a user is difficultto be standardized, and in order to more accurately and quicklyrecognize a digital image obtained by a user, various types ofpreprocessing of the obtained image are required.

(2) First Cropping (Two-Eye Image Cropping)

An image used in predicting a clinical activity score for thyroid eyedisease needs to include a left eye, a right eye, and the surroundingregions.

However, for a more quick and accurate analysis, it is more efficient touse, in an image analysis, an image of only two eyes and the surroundingregions not including several unnecessary regions (for example, theregions corresponding to the nose, the mouth, the forehead,etc.)(hereinafter, referred to as a two-eye image), rather than an imageof the entire face (hereinafter, referred to as a facial image).

Therefore, it is necessary to cut out the image including two eyes (lefteye/right eye) (hereinafter, referred to as a two-eye image) from theimage of the entire face (hereinafter, referred to as a facial image)obtained by the user.

For example, from the facial image obtained by a user shown in FIG.9(a), a two-eye image (the inner region of the quadrangle marked withthe dotted line) as shown in FIG. 9(b) may be obtained. Hereinafter,obtaining a two-eye image from a facial image acquired by a user in thisway is called two-eye image cropping or first cropping.

(3) Necessity of Applying Additional Cropping Methods

The inventors of the present application built a system for predictingscores for five items related to thyroid eye disease through predictionmodels, which will be described later, using a first cropped image(two-eye image) describe above, but it was found that the accuracy ofprediction was low.

The inventors of the present application determined that the accuracy ofprediction was low because the two-eye image included many regionsunnecessary for an analysis, and determined that it is necessary toobtain a more elaborate cropped image. That is, it was determined thatit is more efficient to separately obtain and use the left eye image andthe right eye image as shown in FIG. 10(b) than to use the two-eye imageas shown in FIG. 10(a) in which the left eye and the right eye areincluded in one image.

(4) Necessity of Applying Different Cropping Methods

It has been described that five of the seven items for evaluating aclinical activity score for thyroid eye disease are the items evaluatedaccording to a doctor's observation with the naked eye, of user'seyeballs and the surrounding regions. The five items are as follows.

1) Conjunctival hyperemia (redness of conjunctiva),

2) Conjunctival edema (swelling of conjunctiva),

3) Lacrimal edema (swelling of lacrimal caruncle),

4) Redness of eyelid, and

5) Eyelid edema (swelling of eyelid).

Hereinafter, as will be described later, in order to evaluate a clinicalactivity score for thyroid eye disease, independent prediction modelsprovided in the present application have been applied to the fivesymptoms.

There may be a method of using an image to which different croppingmethods are applied through five independent prediction models, but theinventors of the present application determined that sufficientprediction accuracy could be obtained by applying an image croppingmethod for analyzing a conjunctiva and a lacrimal caruncle and an imagecropping method for analyzing an eyelid.

(5) Second Cropping (Eye-Outline-Based Cropping)

Hereinafter, eye-outline-based cropping (second cropping) will bedescribed. Second cropping may be applied to both a right eye image anda left eye image, but a description will be given based on the case ofobtaining a right eye cropped image for convenience.

Purpose of Second Cropping

Second cropping is to generate an image to be used as an input image ofa model for predicting whether there is redness of a conjunctiva, amodel for predicting whether there is swelling of a conjunctiva, and amodel for predicting whether there is swelling of a lacrimal caruncleamong the prediction models to be described later. Second cropping is togenerate an image in which information on the cornea and the lacrimalcaruncle is maximized and information on the other regions is minimized.

Input Image

Second cropping may be applied to a facial image or a two-eye image(first cropped image).

Detection of Outline of Eye

According to an embodiment, in order to detect the outline of the righteye, the pixels corresponding to the boundary between the upper eyelidand the eyeball and to the boundary between the lower eyelid and theeyeball may be detected. In addition, in order to detect the outline ofthe right eye, the pixels corresponding to the points at which the uppereyelid and the lower eyelid meet may be detected. Furthermore, in orderto detect the outline of the right eye, the pixels corresponding to thelacrimal caruncle may be detected.

According to another embodiment, the outline pixels corresponding to theoutermost part of the outline of the eye may be detected using an eyeoutline segmentation model, which will be described later.

Determination of Maximum Values and Minimum Values of X and YCoordinates of Outline Pixels

Determining the detected pixels, the maximum value X_(max) of the Xcoordinate values, the minimum value X_(min) of the X coordinate values,the maximum value Y_(max) of the Y coordinate values, and the minimumvalue Y_(min) of the Y coordinate values are determined.

FIG. 11 is a diagram illustrating X_(max), X_(min), Y_(max), and Y_(min)of outline pixels.

Cropped Region Determination

On the basis of the determined X_(max), X_(min), Y_(max), and Y_(min) ofthe outline pixels, a quadrangle having the following four points asvertexes is generated, and the region included inside the quadrangle isdetermined as a cropped region.

(X_(min), Y_(max)),

(X_(max), Y_(max)),

(X_(max), Y_(min)), and

(X_(min), Y_(min))

FIG. 12 is a diagram illustrating a second cropped region determined.

As described above, the second cropped region may be determined in thesame manner for the left eye.

Generation of Second Cropped Images

The second cropped regions are determined, and on the basis of thedetermined second cropped regions, as shown in FIG. 13 , second croppedimages (a second right eye cropped image and a second left eye croppedimage) may be generated from the facial image or the two-eye image byusing the pixels included inside the second cropped regions determinedas described above.

Hereinafter, the term “second right eye cropped image” and the term“right eye outline cropped image” may be used interchangeably, and theterm “second left eye cropped image” and the term “left eye outlinecropped image” may be used interchangeably.

In addition, without specific mention hereinbelow, the term “secondcropped image (or outline cropped image)” may mean either a second righteye cropped image or a second left eye cropped image, or may mean bothdepending on the context.

A second cropped image means an image cropped with respect to ‘theoutline of the eye’. A cropped image generated in a method differentfrom the above-described method is referred to as a second cropped image(outline cropped image) if the cropped image is generated such that thetop, bottom, rightmost, and leftmost pixels of ‘the outline of the eye’are included in the cropped region.

In the meantime, X coordinate values and Y coordinate values in thepresent application have different sizes and directions depending on arelative position with respect to a reference point, so the termsmaximum value and minimum value should be understood in a relativesense, but not in an absolute sense. That is, as the position of theorigin of the coordinate system is changed, the maximum value of theabove-described X coordinate value may be the minimum value of the Xcoordinate value in the coordinate system of which the origin ischanged, and the minimum value of the X coordinate value may be themaximum value of the X coordinate value in the coordinate system ofwhich the origin is changed. This may be equally applied to the Ycoordinate value.

(6) Third Cropping (Eyelid-Included Cropping)

Hereinafter, eyelid-included cropping (third cropping) will bedescribed. Third cropping may be applied to both a right eye image and aleft eye image, but a description will be given based on the case ofobtaining a right eye cropped image for convenience.

Purpose of Third Cropping

Third cropping is to generate an image to be used as an input image of amodel for predicting whether there is redness of eyelids and a model forpredicting whether there is swelling of eyelids among the predictionmodels to be described later. Third cropping is to include informationon eyelids in the image. Herein, rather than cropping with only thepixels corresponding to eyelids, it may be better to generate a croppedimage such that all the pixels included in the outline of the eye areincluded. This is because inference and determination are required forcolor values in order to predict whether there is eyelid redness, andthe color values of the pixels corresponding to the iris and/or thewhite of the eye may be used.

Input Image

Third cropping may be applied to a facial image or a two-eye image(first cropped image).

Detection of Outline of Eye, and Determination of Maximum Values andMinimum Values of X and Y Coordinates of Outline Pixels

According to an embodiment, the eye outline detection method describedin second cropping may be applied as it is, or the outline pixelscorresponding to the outermost part of the outline of the eye may bedetected. Determining the detected pixels, the maximum value X_(max) ofthe X coordinate values, the minimum value X_(min) of the X coordinatevalues, the maximum value Y_(max) of the Y coordinate values, and theminimum value Y_(min) of the Y coordinate values may be determined.

Cropped Region Determination #1

On the basis of the determined Y_(max) value and Y_(min) value, a firstexpansion value Y_(e) determined according to a predetermined criterionis added to the Y_(max) value, and a second expansion value Y_(e)′determined according to a predetermined criterion is subtracted from theY_(min) value, and a third cropped region may be determined similarly tothe above-described method of determining the second cropped region.

That is, a quadrangle having the following four points as vertexes isgenerated, and the region included inside the quadrangle is determinedas a third cropped region.

(X_(min), Y_(max)+Y_(e)),

(X_(max), Y_(max)+Y_(e)),

(X_(max), Y_(min)−Y_(e)′) and

(X_(min), Y_(min)−Y_(e)′)

When the third cropped region is determined in this way, more pixelscorresponding to the upper eyelid and the lower eyelid may be includedin the image, compared to the second cropped region.

Herein, the first expansion value and the second expansion value may bethe same, but are not necessarily the same.

In the meantime, the criterion for determining the first expansion valueand the criterion for determining the second expansion value may be thesame, but are not necessarily the same.

The first expansion value and the second expansion value may bedetermined on the basis of the size of the second cropped region. Forexample, the first expansion value and the second expansion value may bedetermined using the number of pixels corresponding to the lengthcalculated by multiplying an expansion percentage by the horizontallength of the second cropped region. As another example, the firstexpansion value and the second expansion value may be determined usingthe number of pixels corresponding to the length calculated bymultiplying an expansion percentage by the vertical length of the secondcropped region.

Herein, the specific percentage may be any one of the following: 10, 11,12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29,30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 50, 51, 52,53, 54, 55, 56, 57, 58, 59, and 60%.

The expansion percentage used in determining the first expansion valueand the expansion percentage used in determining the second expansionvalue may be the same, but are not necessarily the same and may bedifferent from each other.

When the horizontal length of the second cropped region is used indetermining the first expansion value, the horizontal length may also beused in determining the second expansion value, but no limitationthereto is imposed and the vertical length may be used in determiningthe second expansion value.

Cropped Region Determination #2

On the basis of the determined X_(max) value and X_(min) value, a firstwidth expansion value X_(we) is determined according to a predeterminedcriterion, and a second width expansion value X_(we)′ is determinedaccording to a predetermined criterion.

On the basis of the determined Y_(max) value and Y_(min) value, a firstheight expansion value Y_(he) is determined according to a predeterminedcriterion, and a second height expansion value Y_(he)′ is determinedaccording to a predetermined criterion.

On the basis of the value obtained by adding the first width expansionvalue X_(we) to the X_(max) value, the value obtained by subtracting thesecond width expansion value X_(we)′ from the X_(min) value, the valueobtained by adding the first height expansion value Y_(he) to theY_(max) value, and the value obtained by subtracting the second heightexpansion value Y_(he)′ from the Y_(min) value, a third cropped regionmay be determined similarly to the above-described method of determiningthe second cropped region.

That is, a quadrangle having the following four points as vertexes isgenerated, and the region included inside the quadrangle is determinedas a third cropped region.

(X_(min)−X_(we)′, Y_(max)+Y_(he)),

(X_(max)+X_(we), Y_(max)+Y_(he)),

(X_(max)+X_(we), Y_(min)−Y_(he)′) and

(X_(min)−X_(we)′, Y_(min)−Y_(he)′)

When the third cropped region is determined in this way, more pixelscorresponding to the upper eyelid and the lower eyelid may be includedin the image, compared to the second cropped region.

Furthermore, the cropped image includes more pixels in a left-rightdirection than the image cropped by the method “cropped regiondetermination #1”. As a result, the cropped image includes moreinformation on the upper eyelid and the lower eyelid. Because the widthof the upper eyelid and the lower eyelid is generally wider than thewidth of the eyeball exposed to the outside, more pixels correspondingto the upper eyelid and the lower eyelid are included through verticalexpansion as well as horizontal expansion.

Herein, the first width expansion value and the second width expansionvalue may be the same, but are not necessarily the same.

In the meantime, the criterion for determining the first heightexpansion value and the criterion for determining the second heightexpansion value may be the same, but are not necessarily the same.

The first width expansion value and the second width expansion value maybe determined on the basis of the size of the second cropped region. Forexample, the first width expansion value and the second width expansionvalue may be determined using the number of pixels corresponding to thelength calculated by multiplying an expansion percentage by thehorizontal length of the second cropped region. As another example, thefirst width expansion value and the second width expansion value may bedetermined using the number of pixels corresponding to the lengthcalculated by multiplying an expansion percentage by the vertical lengthof the second cropped region.

Herein, the specific percentage may be any one of the following: 10, 11,12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29,30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 50, 51, 52,53, 54, 55, 56, 57, 58, 59, and 60%.

The expansion percentage used in determining the first width expansionvalue and the expansion percentage used in determining the second widthexpansion value may be the same, but are not necessarily the same andmay be different from each other.

When the horizontal length of the second cropped region is used indetermining the first width expansion value, the horizontal length mayalso be used in determining the second width expansion value, but nolimitation thereto is imposed and the vertical length may be used indetermining the second expansion value.

In the meantime, the method of determining the first height expansionvalue and the second height expansion value is the same as theabove-described method of determining the first expansion value and thesecond expansion value, so a detailed description will be omitted.

Generation of Third Cropped Images

The third cropped regions are determined, and on the basis of thedetermined third cropped regions, as shown in FIG. 14 , third croppedimages (a third right eye cropped image and a third left eye croppedimage) may be generated from the facial image or the two-eye image byusing the pixels included inside the second cropped regions determinedas described above.

For reference, FIG. 14(a) shows the third cropped images cropped by theabove-described method ‘cropped region determination #1’, and FIG. 14(b)shows the third cropped images cropped by the above-described method‘cropped region determination #2’.

Hereinafter, the term “third right eye cropped image” and the term“right eyelid-included-cropped image” may be used interchangeably, andthe term “third left eye cropped image” and the term “lefteyelid-included-cropped image” may be used interchangeably.

In addition, without specific mention hereinbelow, the term “thirdcropped image (or eyelid-included cropped image)” may mean either athird right eye cropped image or a third left eye cropped image, or maymean both depending on the context.

A third cropped image means an image that is generated such that theimage includes information on eyelids. A cropped image generated in amethod different from the above-described method is referred to as athird cropped image (eyelid-included cropped image) if the boundary ofthe cropped region is determined such that the pixels corresponding toeyelids are additionally included.

(7) Iris Segmentation

Hereinafter, iris segmentation will be described.

Iris segmentation may be performed by a model that distinguishes aregion corresponding to the iris or cornea from an image of the eyeballand the surroundings.

By using iris segmentation, the pixels corresponding to the iris withinthe image may be inferred as shown in FIG. 15 .

By using iris segmentation, the pixels corresponding to the iris exposedto the outside within the image may be inferred as shown in FIG. 16 .

The model for iris segmentation may receive a facial image as inputdata, may output ‘1’ for the pixels inferred to be the pixelscorresponding to the iris within the facial image, and may output ‘0’for the other pixels.

The iris segmentation model may be trained using training data thatincludes a facial image and an image in which the pixel values of thepixels corresponding to the iris within a facial image are ‘1’ and thepixel values of the remaining pixels are ‘0’.

Although it has been described that a facial image is received as inputdata and is subjected to iris segmentation, iris segmentation may alsobe performed using the above-described two-eye image as input data.

(8) Eye Outline Segmentation

Hereinafter, eye outline segmentation will be described.

Eye outline segmentation may be performed by a model that distinguishesa region corresponding to the inside of the outline of the eye from animage of the eyeball and the surroundings.

By using eye outline segmentation, the pixels corresponding to theinside of the outline of the eye within the image may be inferred asshown in FIG. 17 .

The model for eye outline segmentation may receive a facial image asinput data, may output ‘1’ for the pixels inferred to be the pixelscorresponding to the inside of the outline of the eye within the facialimage, and may output ‘0’ for the other pixels.

The eye outline segmentation model may be trained using training datathat includes a facial image and an image in which the pixel values ofthe pixels corresponding to the inside of the outline of the eye withinthe facial image are ‘1’ and the pixel values of the remaining pixelsare ‘0’.

(9) Masking

First Masking

In the present application, first masking means that in an image,removed is information reflected in the pixel values corresponding tothe regions excluding the pixels corresponding to the conjunctiva andthe lacrimal caruncle.

Removing information reflected in pixel values means changing the pixelvalues of the pixels of which information is to be removed into apredetermined particular value. For example, all the pixel values of thepixels of which information is to be removed may be changed to 0.

First masking may be performed before an image is input to the modelsfor predicting symptoms related to the conjunctiva and the lacrimalcaruncle among the above-described prediction models for predicting aclinical activity score for thyroid eye disease.

Second Masking

In the present application, second masking means that in an image,removed is information reflected in the pixel values corresponding tothe regions corresponding to the cornea exposed to the outside (the irisexposed to the outside).

Second masking may be performed before an image is input to the modelsfor predicting symptoms related to eyelids (upper eyelid and lowereyelid) among the above-described prediction models for predicting aclinical activity score for thyroid eye disease.

Method of First Masking

First masking may be performed on a first masking target image that isone selected from the group of a facial image, a first cropped image(two-eye image), and a second cropped image (outline cropped image).

On the basis of the first masking target image, the eye outlinesegmentation result, and the iris segmentation result, a first maskingimage may be generated. For example, from the first masking targetimage, the values of the pixels excluding the pixels corresponding tothe inside of the outside of the eye, and the pixels values of thepixels corresponding to the iris (or the iris exposed to the outside)may be removed.

FIG. 18 is a diagram illustrating examples of a first masking image.

Method of Second Masking

Second masking may be performed on a second masking target image that isone selected from the group of a facial image, a first cropped image(two-eye image), and a third cropped image (eyelid-included croppedimage).

On the basis of the second masking target image, the eye outlinesegmentation result, and the iris segmentation result, a second maskingimage may be generated. For example, from the second masking targetimage, the pixels values of the pixels corresponding to the inside ofthe outline of the eye and simultaneously corresponding to the iris (orthe iris exposed to the outside) may be removed.

FIG. 19 is a diagram illustrating an example of a second masking image.

Another Embodiment of First Masking

According to the above description, it has been described that firstmasking removes all the pixel values of the pixels corresponding to theregions excluding the pixels corresponding to the conjunctiva and thelacrimal caruncle, but first masking may not remove the pixel values ofthe pixels corresponding to the cornea (iris) when necessary.

However, since colors of irises may vary according to race, removing thepixel values of the pixels corresponding to the iris is advantageous forquicker learning and higher accuracy.

Option of Second Masking

According to the above description, it has been described that secondmasking removes all the pixel values of the pixels corresponding to theiris, but it is allowed not to perform second masking at all.

However, since colors of irises may vary according to race, removing thepixel values of the pixels corresponding to the iris by performingsecond masking is advantageous for quicker learning and higher accuracy.

(10) Lateral Inversion

Necessity of Lateral Inversion

According to a method, which is provided in the present application, ofpredicting a clinical activity score for thyroid eye disease, croppedimages of the left eye and the right eye are used instead of using atwo-eye image.

In the meantime, the outline of an eye is asymmetric. For example, withrespect to the right eye, the lacrimal caruncle is at the left end ofthe right eye, but the point at which the upper eyelid and the lowereyelid meet naturally is at the right end of the right eye.

Accordingly, for quicker learning and more accurate prediction, it ismore effective to distinguish and use a learning model trained withrespect to a right eye and a learning model trained with respect to aleft eye.

However, when the left eye is turned over to be the right eye on thebasis of the line of symmetry between the left eye and the right eye,the shape features of the right eye and the left eye are similar to eachother.

Accordingly, according to the present application, either the right eyeor the left eye is used without lateral inversion, the other eye is usedwith lateral inversion, so that only one learning model can be used.

Lateral Inversion Method

Laterally inverting an image (converting the left and the right of theimage) means that with a left and right reference line (X=a) verticallycrossing the image to be inverted and dividing the image in half leftand right, when a first pixel value corresponds to the pixel (a+Δ, Y) inthe image and a second pixel value corresponds to the pixel (a−Δ, Y),the pixel value of (a+Δ, Y) is changed from the first pixel value to thesecond pixel value and the pixel value of (a−Δ, Y) is changed from thesecond pixel value to the first pixel value.

Lateral Inversion Target Image

Laterally inverting either the image of the left eye or the image of theright eye is sufficient. Which one of the left eye image and the righteye image is subjected to lateral inversion is determined according towhich one of the left eye image and the right eye image is based whenthe prediction models, which will be described later, are trained.

In the meantime, lateral inversion may be performed on the image onwhich both masking and cropping (second cropping or third cropping) havebeen performed, or lateral inversion may be performed on the image onwhich only cropping has been performed, but masking has not beenperformed.

FIGS. 20 to 22 are diagrams illustrating various examples of originalimages and laterally inverted images.

Option of Lateral Inversion

However, as described above, lateral inversion is applied to unify aprediction model for the left eye and a prediction model for the righteye. Therefore, if the prediction model for the left eye and theprediction model for the right eye are realized as different models,lateral inversion preprocessing may be omitted.

(11) Resizing

Necessity of Resizing

As described above, when an image is cropped with respect to the outlineof the eye and the cropped image is used, sizes of eyes vary from personto person and cropped images vary in size from person to person.

In the meantime, when a left eye image and a right eye image areindependently cropped and obtained, the left eye cropped image and theright eye cropped image of the same person are different from each otherbecause of the difference in size between the left eye and the righteye.

For this reason, before an eye image is input to the prediction models,which will be described later, it is necessary to resize the eye imageto standard sizes corresponding to the respective prediction models.

Standard Size for Each Prediction Model

The standard sizes corresponding to a first prediction model to a fifthprediction model, respectively, may be different from each other.

The standard sizes corresponding to the prediction models using a secondcropped image as an input image may be the same.

The standard sizes corresponding to the prediction models using a thirdcropped image as an input image may be the same.

The standard size corresponding to the prediction models using a secondcropped image as an input image may be different from the standard sizecorresponding to the prediction models using a third cropped image as aninput image.

Alternatively, the standard sizes corresponding to the first predictionmodel to the fifth prediction model, respectively, may be the same.

Resizing Method

The size of a resizing target image is adjusted to a standard size.

When the width or height of the resizing target image is greater thanthe width or height of the standard size, the width or height of theresizing target image may be decreased.

When the width or height of the resizing target image is less than thewidth or height of the standard size, the width or height of theresizing target image may be increased.

In resizing, the aspect ratio of the image before resizing may bedifferent from the aspect ratio of the image after resizing.

4. Prediction Models

(1) First Prediction Model

Purpose and Operation of First Prediction Model

The first prediction model is a model for predicting whether there isconjunctival hyperemia.

The first prediction model may receive an eye image as input data andmay output a probability value that the conjunctiva captured in theinput eye image is hyperemic.

When the first prediction model includes a first left eye predictionmodel and a first right eye prediction model, the first left eyeprediction model may receive a left eye image and output a probabilityvalue that the conjunctiva captured in the left eye image is hyperemic,and the first right eye prediction model may receive a right eye imageand may output a probability value that the conjunctiva captured in theright eye image is hyperemic.

When the first prediction model is not dualized and is realized as onemodel, the first prediction model may receive either a right eye imageor a left eye image to output a probability value that the conjunctivacaptured in the input image is hyperemic, and may receive the otherimage to output a probability value that the conjunctiva captured in theinput image is hyperemic.

The eye image may be an image preprocessed by the above-describedpreprocessing algorithms.

For example, the eye image may be an image on which preprocessingaccording to second cropping is performed.

As another example, the eye image may be an image on which preprocessingincluding second cropping and resizing is performed.

As still another example, the eye image may be an image on whichpreprocessing including second cropping, first masking, and resizing isperformed.

As yet still another example, the eye image may be an image on whichpreprocessing including second cropping, first masking, lateralinversion, and resizing is performed.

In the present specification, the first prediction model may be called aconjunctival hyperemia prediction model.

Training of First Prediction Model

To train the first prediction model, a plurality of training data setsmay be prepared. A training data set may include an eye image and anevaluation value for conjunctival hyperemia captured in the eye image.The eye image may be an image preprocessed by the above-describedpreprocessing algorithms. For example, the eye image may be an image onwhich preprocessing including second cropping, first masking, andresizing is performed.

To train the first prediction model, an artificial intelligence modelmay be prepared.

Examples of the artificial intelligence model may be a support-vectormachine (SVM), Random Forest, Gradient Boosting Algorithm, ResNet, VGG,GoogT eNet, MobileNet, and Vision Transformer.

Next, the eye images included in the prepared plurality of training datasets are input to the artificial intelligence model, and training isperformed using the evaluation value corresponding to each of the inputeye images and an output value output from the artificial intelligencemodel.

When the first prediction model includes the first left eye predictionmodel and the first right eye prediction model, the plurality oftraining data sets for training the first left eye prediction model mayinclude left eye images and evaluation values for conjunctival hyperemiacaptured in the left eye images, and the plurality of training data setsfor training a first right eye prediction model may include right eyeimages and evaluation values for conjunctival hyperemia captured in theright eye images. In the meantime, in order to increase the number oftraining data sets, the plurality of training data sets for training thefirst left eye prediction model may include right eye images on whichlateral inversion is processed and evaluation values for conjunctivalhyperemia captured in the right eye images, and the plurality oftraining data sets for training the first right eye prediction model mayinclude left eye images on which lateral inversion is processed andevaluation values for conjunctival hyperemia captured in the left eyeimages.

When it is intended not to dualize the first prediction model, but torealize the first prediction model as one model, the plurality oftraining data sets may include right eye images and evaluation valuesfor conjunctival hyperemia captured in the right eye images, or mayinclude left eye images on which lateral inversion is performed andevaluation values for conjunctival hyperemia captured in the left eyeimages. Alternatively, the plurality of training data sets may includeleft eye images and evaluation values for conjunctival hyperemiacaptured in the left eye images, or may include right eye images onwhich lateral inversion is performed and evaluation values forconjunctival hyperemia captured in the right eye images.

In the meantime, in training the first prediction model, in order topredict whether there is conjunctival hyperemia without distinguishingbetween right eye images and left eye images, all right eye images,right eye images on which lateral inversion is performed, left eyeimages, and left eye images on which lateral inversion is performed areused as training data for training one model.

For example, when the first prediction model includes the first left eyeprediction model and the first right eye prediction model, the pluralityof training data sets for training the first left eye prediction modelmay include: left eye images and evaluation values for conjunctivalhyperemia captured in the left eye images; and right eye images on whichlateral inversion is performed and evaluation values for conjunctivalhyperemia captured in the right eye images, and the plurality oftraining data sets for training the first right eye prediction model mayinclude: right eye images and evaluation values for conjunctivalhyperemia captured in the right eye images; and left eye images on whichlateral inversion is performed and evaluation values for conjunctivalhyperemia captured in the left eye images.

When it is intended not to dualize the first prediction model, but torealize the first prediction model as one model, the plurality oftraining data sets may include: right eye images and evaluation valuesfor conjunctival hyperemia captured in the right eye images; right eyeimages on which lateral inversion is performed and evaluation values forconjunctival hyperemia captured in the right eye images; left eye imagesand evaluation values for conjunctival hyperemia captured in the lefteye images; and left eye images on which lateral inversion is performedand evaluation values for conjunctival hyperemia captured in the lefteye images.

(2) Second Prediction Model

Purpose and Operation of Second Prediction Model

The second prediction model is a model for predicting whether there isconjunctival edema.

The second prediction model may receive an eye image as input data andmay output a probability value of the presence of conjunctival edemacaptured in the input eye image.

When the second prediction model includes a second left eye predictionmodel and a second right eye prediction model, the second left eyeprediction model may receive a left eye image and output a probabilityvalue of the presence of conjunctival edema captured in the left eyeimage, and the second right eye prediction model may receive a right eyeimage and output a probability value of the presence of conjunctivaledema captured in the right eye image.

When the second prediction model is not dualized and is realized as onemodel, the second prediction model may receive either a right eye imageor a left eye image to output a probability value of the presence ofconjunctival edema captured in the input image, and may receive theother image to output a probability value of the presence ofconjunctival edema captured in the input image.

The eye image may be an image preprocessed by the above-describedpreprocessing algorithms.

For example, the eye image may be an image on which preprocessingaccording to second cropping is performed.

As another example, the eye image may be an image on which preprocessingincluding second cropping and resizing is performed.

As still another example, the eye image may be an image on whichpreprocessing including second cropping, first masking, and resizing isperformed.

As yet still another example, the eye image may be an image on whichpreprocessing including second cropping, first masking, lateralinversion, and resizing is performed.

In the present specification, the second prediction model may be calleda conjunctival edema prediction model.

Training of Second Prediction Model

To train the second prediction model, a plurality of training data setsmay be prepared. A training data set may include an eye image and anevaluation value for the presence of conjunctival edema captured in theeye image. The eye image may be an image preprocessed by theabove-described preprocessing algorithms. For example, the eye image maybe an image on which preprocessing including second cropping, firstmasking, and resizing is performed.

To train the second prediction model, an artificial intelligence modelmay be prepared.

Examples of the artificial intelligence model may be a support-vectormachine (SVM), Random Forest, Gradient Boosting Algorithm, ResNet, VGG,GoogT eNet, MobileNet, and Vision Transformer.

Next, the eye images included in the prepared plurality of training datasets are input to the artificial intelligence model, and training isperformed using the evaluation value corresponding to each of the inputeye images and an output value output from the artificial intelligencemodel.

When the second prediction model includes the second left eye predictionmodel and the second right eye prediction model, the plurality oftraining data sets for training the second left eye prediction model mayinclude left eye images and evaluation values for the presence ofconjunctival edema captured in the left eye images, and the plurality oftraining data sets for training the second right eye prediction modelmay include right eye images and evaluation values for the presence ofconjunctival edema captured in the right eye images. In the meantime, inorder to increase the number of training data sets, the plurality oftraining data sets for training the second left eye prediction model mayinclude right eye images on which lateral inversion is processed andevaluation values for the presence of conjunctival edema captured in theright eye images, and the plurality of training data sets for trainingthe second right eye prediction model may include left eye images onwhich lateral inversion is processed and evaluation values for thepresence of conjunctival edema captured in the left eye images.

When it is intended not to dualize the second prediction model, but torealize the second prediction model as one model, the plurality oftraining data sets may include right eye images and evaluation valuesfor the presence of conjunctival edema captured in the right eye images,or may include left eye images on which lateral inversion is performedand evaluation values for the presence of conjunctival edema captured inthe left eye images. Alternatively, the plurality of training data setsmay include left eye images and evaluation values for the presence ofconjunctival edema captured in the left eye images, or may include righteye images on which lateral inversion is performed and evaluation valuesfor the presence of conjunctival edema captured in the right eye images.

In the meantime, in training the second prediction model, in order topredict whether there is conjunctival edema without distinguishingbetween right eye images and left eye images, all right eye images,right eye images on which lateral inversion is performed, left eyeimages, and left eye images on which lateral inversion is performed areused as training data for training one model.

For example, when the second prediction model includes the second lefteye prediction model and the second right eye prediction model, theplurality of training data sets for training the second left eyeprediction model may include: left eye images and evaluation values forconjunctival edema captured in the left eye images; and right eye imageson which lateral inversion is performed and evaluation values forconjunctival edema captured in the right eye images, and the pluralityof training data sets for training the second right eye prediction modelmay include: right eye images and evaluation values for conjunctivaledema captured in the right eye images; and left eye images on whichlateral inversion is performed and evaluation values for conjunctivaledema captured in the left eye images.

When it is intended not to dualize the second prediction model, but torealize the second prediction model as one model, the plurality oftraining data sets may include: right eye images and evaluation valuesfor conjunctival edema captured in the right eye images; right eyeimages on which lateral inversion is performed and evaluation values forconjunctival edema captured in the right eye images; left eye images andevaluation values for conjunctival edema captured in the left eyeimages; and left eye images on which lateral inversion is performed andevaluation values for conjunctival edema captured in the left eyeimages.

(3) Third Prediction Model

Purpose and Operation of Third Prediction Model

The third prediction model is a model for predicting whether there islacrimal edema.

The third prediction model may receive an eye image as input data andmay output a probability value of the presence of lacrimal edemacaptured in the input eye image.

When the third prediction model includes a third left eye predictionmodel and a third right eye prediction model, the third left eyeprediction model may receive a left eye image and output a probabilityvalue of the presence of lacrimal edema captured in the left eye image,and the third right eye prediction model may receive a right eye imageand output a probability value of the presence of lacrimal edemacaptured in the right eye image.

When the third prediction model is not dualized and is realized as onemodel, the third prediction model may receive either a right eye imageor a left eye image to output a probability value of the presence oflacrimal edema captured in the input image, and may receive the otherimage to output a probability value of the presence of lacrimal edemacaptured in the input image.

The eye image may be an image preprocessed by the above-describedpreprocessing algorithms.

For example, the eye image may be an image on which preprocessingaccording to second cropping is performed.

As another example, the eye image may be an image on which preprocessingincluding second cropping and resizing is performed.

As still another example, the eye image may be an image on whichpreprocessing including second cropping, first masking, and resizing isperformed.

As yet still another example, the eye image may be an image on whichpreprocessing including second cropping, first masking, lateralinversion, and resizing is performed.

In the present specification, the third prediction model may be called alacrimal edema prediction model.

Training of Third Prediction Model

To train the third prediction model, a plurality of training data setsmay be prepared. A training data set may include an eye image and anevaluation value for the presence of lacrimal edema captured in the eyeimage. The eye image may be an image preprocessed by the above-describedpreprocessing algorithms. For example, the eye image may be an image onwhich preprocessing including second cropping, first masking, andresizing is performed.

To train the third prediction model, an artificial intelligence modelmay be prepared.

Examples of the artificial intelligence model may be a support-vectormachine (SVM), Random Forest, Gradient Boosting Algorithm, ResNet, VGG,GoogT eNet, MobileNet, and Vision Transformer.

Next, the eye images included in the prepared plurality of training datasets are input to the artificial intelligence model, and training isperformed using the evaluation value corresponding to each of the inputeye images and an output value output from the artificial intelligencemodel.

When the third prediction model includes the third left eye predictionmodel and the third right eye prediction model, the plurality oftraining data sets for training the third left eye prediction model mayinclude left eye images and evaluation values for the presence oflacrimal edema captured in the left eye images, and the plurality oftraining data sets for training the third right eye prediction model mayinclude right eye images and evaluation values for the presence oflacrimal edema captured in the right eye images. In the meantime, inorder to increase the number of training data sets, the plurality oftraining data sets for training the third left eye prediction model mayinclude right eye images on which lateral inversion is processed andevaluation values for the presence of lacrimal edema captured in theright eye images, and the plurality of training data sets for trainingthe third right eye prediction model may include left eye images onwhich lateral inversion is processed and evaluation values for thepresence of lacrimal edema captured in the left eye images.

When it is intended not to dualize the third prediction model, but torealize the third prediction model as one model, the plurality oftraining data sets may include right eye images and evaluation valuesfor the presence of lacrimal edema captured in the right eye images, ormay include left eye images on which lateral inversion is performed andevaluation values for the presence of lacrimal edema captured in theleft eye images. Alternatively, the plurality of training data sets mayinclude left eye images and evaluation values for the presence oflacrimal edema captured in the left eye images, or may include right eyeimages on which lateral inversion is performed and evaluation values forthe presence of lacrimal edema captured in the right eye images.

In the meantime, in training the third prediction model, in order topredict whether there is lacrimal edema without distinguishing betweenright eye images and left eye images, all right eye images, right eyeimages on which lateral inversion is performed, left eye images, andleft eye images on which lateral inversion is performed are used astraining data for training one model.

For example, when the third prediction model includes the third left eyeprediction model and the third right eye prediction model, the pluralityof training data sets for training the third left eye prediction modelmay include: left eye images and evaluation values for lacrimal edemacaptured in the left eye images; and right eye images on which lateralinversion is performed and evaluation values for lacrimal edema capturedin the right eye images, and the plurality of training data sets fortraining the third right eye prediction model may include: right eyeimages and evaluation values for lacrimal edema captured in the righteye images; and left eye images on which lateral inversion is performedand evaluation values for lacrimal edema captured in the left eyeimages.

When it is intended not to dualize the third prediction model, but torealize the third prediction model as one model, the plurality oftraining data sets may include: right eye images and evaluation valuesfor lacrimal edema captured in the right eye images; right eye images onwhich lateral inversion is performed and evaluation values for lacrimaledema captured in the right eye images; left eye images and evaluationvalues for lacrimal edema captured in the left eye images; and left eyeimages on which lateral inversion is performed, and evaluation valuesfor lacrimal edema captured in the left eye images.

(4) FOURTH PREDICTION MODEL

Purpose and Operation of Fourth Prediction Model

The fourth prediction model is a model for predicting whether there iseyelid redness.

The fourth prediction model may receive an eye image as input data andmay output a probability value of the presence of eyelid rednesscaptured in the input eye image.

When the fourth prediction model includes a fourth left eye predictionmodel and a fourth right eye prediction model, the fourth left eyeprediction model may receive a left eye image and output a probabilityvalue of the presence of eyelid redness captured in the left eye image,and the fourth right eye prediction model may receive a right eye imageand output a probability value of the presence of eyelid rednesscaptured in the right eye image.

When the fourth prediction model is not dualized and is realized as onemodel, the fourth prediction model may receive either a right eye imageor a left eye image to output a probability value of the presence ofeyelid redness captured in the input image, and may receive the otherimage to output a probability value of the presence of eyelid rednesscaptured in the input image.

The eye image may be an image preprocessed by the above-describedpreprocessing algorithms.

For example, the eye image may be an image on which preprocessingaccording to third cropping is performed.

As another example, the eye image may be an image on which preprocessingincluding third cropping and resizing is performed.

As still another example, the eye image may be an image on whichpreprocessing including third cropping, lateral inversion, and resizingis performed.

As still another example, the eye image may be an image on whichpreprocessing including third cropping, second masking, and resizing isperformed.

As yet still another example, the eye image may be an image on whichpreprocessing including third cropping, second masking, lateralinversion, and resizing is performed.

In the present specification, the fourth prediction model may be calledan eyelid redness prediction model.

Training of Fourth Prediction Model

To train the fourth prediction model, a plurality of training data setsmay be prepared. A training data set may include an eye image and anevaluation value for eyelid redness captured in the eye image. The eyeimage may be an image preprocessed by the above-described preprocessingalgorithms. For example, the eye image may be an image on whichpreprocessing including second cropping, first masking, and resizing isperformed.

To train the fourth prediction model, an artificial intelligence modelmay be prepared.

Examples of the artificial intelligence model may be a support-vectormachine (SVM), Random Forest, Gradient Boosting Algorithm, ResNet, VGG,GoogT eNet, MobileNet, and Vision Transformer.

Next, the eye images included in the prepared plurality of training datasets are input to the artificial intelligence model, and training isperformed using the evaluation value corresponding to each of the inputeye images and an output value output from the artificial intelligencemodel.

When the fourth prediction model include the fourth left eye predictionmodel and the fourth right eye prediction model, the plurality oftraining data sets for training the fourth left eye prediction model mayinclude left eye images and evaluation values for eyelid rednesscaptured in the left eye images, and the plurality of training data setsfor training the fourth right eye prediction model may include right eyeimages and evaluation values for eyelid redness captured in the righteye images. In the meantime, in order to increase the number of trainingdata sets, the plurality of training data sets for training the fourthleft eye prediction model may include right eye images on which lateralinversion is processed and evaluation values for eyelid redness capturedin the right eye images, and the plurality of training data sets fortraining the fourth right eye prediction model may include left eyeimages on which lateral inversion is processed and evaluation values foreyelid redness captured in the left eye images.

When it is intended not to dualize the fourth prediction model, but torealize the fourth prediction model as one model, the plurality oftraining data sets may include right eye images and evaluation valuesfor eyelid redness captured in the right eye images, or may include lefteye images on which lateral inversion is performed and evaluation valuesfor eyelid redness captured in the left eye images. Alternatively, theplurality of training data sets may include left eye images andevaluation values for eyelid redness captured in the left eye images, ormay include right eye images on which lateral inversion is performed andevaluation values for eyelid redness captured in the right eye images.

In the meantime, in training the fourth prediction model, in order topredict whether there is eyelid redness without distinguishing betweenright eye images and left eye images, all right eye images, right eyeimages on which lateral inversion is performed, left eye images, andleft eye images on which lateral inversion is performed are used astraining data for training one model.

For example, when the fourth prediction model includes the fourth lefteye prediction model and the fourth right eye prediction model, theplurality of training data sets for training the fourth left eyeprediction model may include: left eye images and evaluation values foreyelid redness captured in the left eye images; and right eye images onwhich lateral inversion is performed and evaluation values for eyelidredness captured in the right eye images, and the plurality of trainingdata sets for training the fourth right eye prediction model mayinclude: right eye images and evaluation values for eyelid rednesscaptured in the right eye images; and left eye images on which lateralinversion is performed and evaluation values for eyelid redness capturedin the left eye images.

When it is intended not to dualize the fourth prediction model, but torealize the fourth prediction model as one model, the plurality oftraining data sets may include: right eye images and evaluation valuesfor eyelid redness captured in the right eye images; right eye images onwhich lateral inversion is performed and evaluation values for eyelidredness captured in the right eye images; left eye images and evaluationvalues for eyelid redness captured in the left eye images; and left eyeimages on which lateral inversion is performed and evaluation values foreyelid redness captured in the left eye images.

(5) Fifth Prediction Model

Purpose and Operation of Fifth Prediction Model

The fifth prediction model is a model for predicting whether there iseyelid edema.

The fifth prediction model may receive an eye image as input data andmay output a probability value of the presence of eyelid edema capturedin the input eye image.

When the fifth prediction model includes a fifth left eye predictionmodel and a fifth right eye prediction model, the fifth left eyeprediction model may receive a left eye image and output a probabilityvalue of the presence of eyelid edema captured in the left eye image,and the fifth right eye prediction model may receive a right eye imageand output a probability value of the presence of eyelid edema capturedin the right eye image.

When the fifth prediction model is not dualized and is realized as onemodel, the fifth prediction model may receive either a right eye imageor a left eye image to output a probability value of the presence ofeyelid edema captured in the input image, and may receive the otherimage to output a probability value of the presence of eyelid edemacaptured in the input image.

The eye image may be an image preprocessed by the above-describedpreprocessing algorithms.

For example, the eye image may be an image on which preprocessingaccording to third cropping is performed.

As another example, the eye image may be an image on which preprocessingincluding third cropping and resizing is performed.

As still another example, the eye image may be an image on whichpreprocessing including third cropping, lateral inversion, and resizingis performed.

As still another example, the eye image may be an image on whichpreprocessing including third cropping, second masking, and resizing isperformed.

As yet still another example, the eye image may be an image on whichpreprocessing including third cropping, second masking, lateralinversion, and resizing is performed.

In the present specification, the fifth prediction model may be calledan eyelid edema prediction model.

Training of Fifth Prediction Model

To train the fifth prediction model, a plurality of training data setsmay be prepared. A training data set may include an eye image and anevaluation value for the presence of eyelid edema captured in the eyeimage. The eye image may be an image preprocessed by the above-describedpreprocessing algorithms. For example, the eye image may be an image onwhich preprocessing including third cropping, second masking, andresizing is performed.

To train the fifth prediction model, an artificial intelligence modelmay be prepared.

Examples of the artificial intelligence model may be a support-vectormachine (SVM), Random Forest, Gradient Boosting Algorithm, ResNet, VGG,GoogT eNet, MobileNet, and Vision Transformer.

Next, the eye images included in the prepared plurality of training datasets are input to the artificial intelligence model, and training isperformed using the evaluation value corresponding to each of the inputeye images and an output value output from the artificial intelligencemodel.

When the fifth prediction model includes the fifth left eye predictionmodel and the fifth right eye prediction model, the plurality oftraining data sets for training the fifth left eye prediction model mayinclude left eye images and evaluation values for the presence of eyelidedema captured in the left eye images, and the plurality of trainingdata sets for training the fifth right eye prediction model may includeright eye images and evaluation values for the presence of eyelid edemacaptured in the right eye images. In the meantime, in order to increasethe number of training data sets, the plurality of training data setsfor training the fifth left eye prediction model may include right eyeimages on which lateral inversion is processed and evaluation values forthe presence of eyelid edema captured in the right eye images, and theplurality of training data sets for training the fifth right eyeprediction model may include left eye images on which lateral inversionis processed and evaluation values for the presence of eyelid edemacaptured in the left eye images.

When it is intended not to dualize the fifth prediction model, but torealize the fifth prediction model as one model, the plurality oftraining data sets may include right eye images and evaluation valuesfor the presence of eyelid edema captured in the right eye images, ormay include left eye images on which lateral inversion is performed andevaluation values for the presence of eyelid edema captured in the lefteye images. Alternatively, the plurality of training data sets mayinclude left eye images and evaluation values for the presence of eyelidedema captured in the left eye images, or may include right eye imageson which lateral inversion is performed and evaluation values for thepresence of eyelid edema captured in the right eye images.

In the meantime, in training the fifth prediction model, in order topredict whether there is eyelid edema without distinguishing betweenright eye images and left eye images, all right eye images, right eyeimages on which lateral inversion is performed, left eye images, andleft eye images on which lateral inversion is performed are used astraining data for training one model.

For example, when the fifth prediction model includes the fifth left eyeprediction model and the fifth right eye prediction model, the pluralityof training data sets for training the fifth left eye prediction modelmay include: left eye images and evaluation values for eyelid edemacaptured in the left eye images; and right eye images on which lateralinversion is performed and evaluation values for eyelid edema capturedin the right eye images, and the plurality of training data sets fortraining the fifth right eye prediction model may include: right eyeimages and evaluation values for eyelid edema captured in the right eyeimages; and left eye images on which lateral inversion is performed andevaluation values for eyelid edema captured in the left eye images.

When it is intended not to dualize the fifth prediction model, but torealize the fifth prediction model as one model, the plurality oftraining data sets may include: right eye images and evaluation valuesfor eyelid edema captured in the right eye images; right eye images onwhich lateral inversion is performed and evaluation values for eyelidedema captured in the right eye images; left eye images and evaluationvalues for eyelid edema captured in the left eye images; and left eyeimages on which lateral inversion is performed and evaluation values foreyelid edema captured in the left eye images.

The training of the prediction models may be performed by an electronicdevice, and in particular, may be performed by the server 20 describedabove. Furthermore, the training of the prediction models by theelectronic device or the server 20 means a series of processes forenabling output values of the prediction models for input data to bevalues similar to output values labelled with the input data. To thisend, the electronic device or the server 20 may use the differencesbetween the output values of the prediction models and the labelledvalues to change a weight value of each of the nodes included in theprediction models. Herein, the electronic device or the server 20 maydetermine the amount of change in the weight value of each of the nodesby using various feedback functions.

Hereinafter, the following methods through the above-described system 1will be described: a method of predicting each symptom related to aclinical activity score for thyroid eye disease by preprocessing an eyeimage and inputting the preprocessed eye image to the above-describedprediction models; a method of predicting a clinical activity score onthe basis of a prediction result for each symptom; and a method ofmonitoring a prediction result of a clinical activity score and givingguidance or recommendation according to the monitored result so that auser visits the hospital and has a medical examination.

5. Conjunctival Hyperemia Prediction Method

The conjunctival hyperemia prediction method described in the presentapplication may be performed by the server 20.

FIG. 23 is a flowchart illustrating a conjunctival hyperemia predictionmethod.

Referring to FIG. 23 , the server 20 acquires a facial image in stepS100, preprocesses the acquired facial image in step S110, inputs thepreprocessed image to the above-described first prediction model(conjunctival hyperemia prediction model) in step S120, and acquires anoutput value of the first prediction model in step S130.

Acquisition of Facial Image

The server 20 acquires a facial image in step S100. The server 20 mayacquire the facial image from the user terminal 10.

Preprocessing of Facial Image

The server 20 may preprocess the acquired facial image in step S110. Theserver 20 may perform, on the acquired facial image, iris segmentation,eye outline segmentation, masking, cropping, and resizing, which aredescribed above.

Segmentation Processing

The server 20 may perform the iris segmentation and the eye outlinesegmentation, and the server 20 may thus determine the pixelscorresponding to the iris and the pixels corresponding to the inside ofthe outline of the eye within the acquired facial image. The server 20may determine the coordinate values of the pixels corresponding to theiris and the coordinate values of the pixels corresponding to the insideof the outline of the eye.

Masking Processing

The server 20 may perform the first masking on the facial image on thebasis of information on the determined pixels. Through the first maskingprocessing, the server 20 may remove the pixel values of the pixelsexcluding the pixels corresponding to the conjunctiva exposed to theoutside and the lacrimal caruncle among the pixels included in thefacial image. Accordingly, the pixel values of the pixels correspondingto the conjunctiva and lacrimal caruncle of the left eye and to theconjunctiva and lacrimal caruncle of the right eye may remain to be theoriginal pixel values, but the pixel values of the pixels correspondingto the iris (or cornea) of the left eye, to the iris (or cornea) of theright eye, to the outside of the outline of the left eye, and to theoutside of the outline of the right eye may be removed or changed toother values.

Cropping Processing

The server 20 may crop the masked facial image. The server 20 may cropthe masked facial image to generate a left eye cropped image and a righteye cropped image. In performing the conjunctival hyperemia predictionmethod, the server 20 may use the second cropping (eye outline cropping)method among the cropping methods described above. Since the secondcropping method has already been described in detail, a detaileddescription thereof will be omitted here.

Resizing Processing and Lateral Inversion Processing

The server 20 may resize the size of the left eye cropped image and thesize of the right eye cropped image to a predetermined size.

In the meantime, when the first prediction model is not dualized, butrealized as one model, the server 20 may laterally invert either theleft eye cropped image or the right eye cropped image as describedabove. The server 20 does not laterally invert the other among the lefteye cropped image and the right eye cropped image. Herein, it isdetermined that the criterion for determining which one of the left eyeimage and the right eye image is subjected to lateral inversion is thesame as the criterion applied when the first prediction model istrained. That is, in training the first prediction model, when the lefteye image is inverted and the right eye image is not inverted, theserver 20 inverts the left eye image and does not invert the right eyeimage, similarly.

As described above, in realizing the first prediction model, when thefirst prediction model is dualized to the first left eye predictionmodel and the first right eye prediction model, the server 20 may notperform lateral inversion processing.

In the meantime, it has been described that when preprocessing isperformed, segmentation, masking processing, cropping processing,resizing processing, and lateral inversion processing are performed, butthe sequence of these types of preprocessing may be changed within arange capable of achieving the purpose of the conjunctival hyperemiaprediction method disclosed in the present application.

Input of Preprocessed Image

The server 20 may input the preprocessed image to the first predictionmodel in step S120.

When the first prediction model is not dualized and is realized as onemodel, the server 20 inputs the right eye preprocessed image and thelaterally inverted left eye preprocessed image to the first predictionmodel in order.

In realizing the first prediction model, when the first prediction modelis dualized to the first left eye prediction model and the first righteye prediction model, the server 20 inputs the left eye preprocessedimage to the first left eye prediction model and inputs the right eyepreprocessed image to the first right eye prediction model.Alternatively, the server 20 may input the left eye preprocessed imageto the first left eye prediction model, may input the laterally invertedleft eye preprocessed image to the first right eye prediction model, mayinput the right eye preprocessed image to the first right eye predictionmodel, and may input the laterally inverted right eye preprocessed imageto the first left eye prediction model.

In realizing the first prediction model, when the first prediction modelis not dualized and is realized as one model and, simultaneously, istrained to be capable of determining whether there is conjunctivalhyperemia without distinguishing between a left eye image and a righteye image, the server 20 may input the left eye preprocessed image andthe right eye preprocessed image to the first prediction model withoutlateral inversion. Alternatively, the server 20 may input the left eyepreprocessed image, the laterally inverted left eye preprocessed image,the right eye preprocessed image, and the laterally inverted right eyepreprocessed image to the first prediction model.

Conjunctival Hyperemia Prediction Result

The server 20 may output a result value output from the first predictionmodel in step S130. The result value may be a probability value that ispredicted with respect to conjunctival hyperemia captured in an image.On the basis of a predetermined threshold value, when the predictedprobability value is equal to or greater than the threshold value, theserver 20 determines that the conjunctiva is hyperemic, or when thepredicted probability value is less than the threshold value, the server20 determines that the conjunctiva is not hyperemic.

The server 20 may acquire both a prediction result for the left eye anda prediction result for the right eye.

When the server 20 inputs the left eye preprocessed image to the firstleft eye prediction model, inputs the laterally inverted left eyepreprocessed image to the first right eye prediction model, inputs theright eye preprocessed image to the first right eye prediction model,and inputs the laterally inverted right eye preprocessed image to thefirst left eye prediction model, the server 20 may obtain a predictionresult for the left eye considering both a result obtained by inputtingthe left eye preprocessed image to the first left eye prediction modeland a result obtained by inputting the laterally inverted left eyepreprocessed image to the first right eye prediction model. Herein, theserver 20 may obtain a prediction result for the right eye consideringboth a result obtained by inputting the right eye preprocessed image tothe first right eye prediction model and a result obtained by inputtingthe laterally inverted right eye preprocessed image to the first lefteye prediction model.

For example, the server 20 may obtain a prediction result for the lefteye on the basis of whether an average value of the result obtained byinputting the left eye preprocessed image to the first left eyeprediction model and the result obtained by inputting the laterallyinverted left eye preprocessed image to the first right eye predictionmodel is equal to or greater than the threshold value.

As another example, when a value of either the result obtained byinputting the left eye preprocessed image to the first left eyeprediction model or the result obtained by inputting the laterallyinverted left eye preprocessed image to the first right eye predictionmodel is equal to or greater than the above-described threshold value,the server 20 may predict that the conjunctiva of the left eye ishyperemic.

As still another example, when both the result obtained by inputting theleft eye preprocessed image to the first left eye prediction model andthe result obtained by inputting the laterally inverted left eyepreprocessed image to the first right eye prediction model are equal toor greater than the above-described threshold value, the server 20 maypredict that the conjunctiva of the left eye is hyperemic.

When the server 20 inputs the left eye preprocessed image, the laterallyinverted left eye preprocessed image, the right eye preprocessed image,and the laterally inverted right eye preprocessed image to the firstprediction model that is not dualized, the server 20 may obtain aprediction result for the left eye considering both a result obtained byinputting the left eye preprocessed image to the first prediction modeland a result obtained by inputting the laterally inverted left eyepreprocessed image to the first prediction model. Herein, the server 20may obtain a prediction result for the right eye considering both aresult obtained by inputting the right eye preprocessed image to thefirst prediction model and a result obtained by inputting the laterallyinverted right eye preprocessed image to the first prediction model.

For example, the server 20 may obtain a prediction result for the lefteye on the basis of whether an average value of the result obtained byinputting the left eye preprocessed image to the first prediction modeland the result obtained by inputting the laterally inverted left eyepreprocessed image to the first prediction model is equal to or greaterthan the threshold value.

As another example, when a value of either the result obtained byinputting the left eye preprocessed image to the first prediction modelor the result obtained by inputting the laterally inverted left eyepreprocessed image to the first prediction model is equal to or greaterthan the above-described threshold value, the server 20 may predict thatthe conjunctiva of the left eye is hyperemic.

As still another example, when both the result obtained by inputting theleft eye preprocessed image to the first prediction model and the resultobtained by inputting the laterally inverted left eye preprocessed imageto the first prediction model are equal to or greater than theabove-described threshold value, the server 20 may predict that theconjunctiva of the left eye is hyperemic.

The above-described method may be similarly applied in determiningwhether there is conjunctival hyperemia of the right eye.

6. Conjunctival Edema Prediction Method

The conjunctival edema prediction method described in the presentapplication may be performed by the server 20.

FIG. 24 is a flowchart illustrating a conjunctival edema predictionmethod.

Referring to FIG. 24 , the server 20 acquires a facial image in stepS200, preprocesses the acquired facial image in step S210, inputs thepreprocessed image to the above-describe second prediction model(conjunctival edema prediction model) in step S220, and acquires anoutput value of the second prediction model in step S230.

The conjunctival edema prediction method is the same as or very similarto the conjunctival hyperemia prediction method except that the secondprediction model is used instead of the first prediction model and afinally acquired result value is a predicted value for whether there isconjunctival edema, so a detailed description of the conjunctival edemaprediction method will be omitted.

7. Lacrimal Edema Prediction Method

The lacrimal edema prediction method described in the presentapplication may be performed by the server 20.

FIG. 25 is a flowchart illustrating a lacrimal edema prediction method.

Referring to FIG. 25 , the server 20 acquires a facial image in stepS300, preprocesses the acquired facial image in step S310, inputs thepreprocessed image to the above-described third prediction model(lacrimal edema prediction model) in step S320, and acquires an outputvalue of the third prediction model in step S330.

The lacrimal edema prediction method is the same as or very similar tothe conjunctival hyperemia prediction method except that the thirdprediction model is used instead of the first prediction model, so adetailed description of the conjunctival edema prediction method will beomitted.

As described above, the conjunctival hyperemia prediction method, theconjunctival edema prediction method, and the lacrimal edema predictionmethod use the same image preprocessing methods, but only the predictionmodels to which the preprocessed images are input are different fromeach other. Therefore, after image preprocessing as described above, theimages may be input to different prediction models.

However, it has been described that the lacrimal edema prediction methoduses the same image preprocessing methods as the conjunctival hyperemiaand conjunctival edema prediction methods, but in some cases, thelacrimal edema prediction method may use the images preprocessed in adifferent way. For example, a preprocessed image may be used that iscropped such that the image includes a lacrimal caruncle and part of aniris. Alternatively, a preprocessed image may be used that is croppedsuch that the image does not include an iris, but includes a lacrimalcaruncle.

8. Eyelid Redness Prediction Method

The eyelid redness prediction method described in the presentapplication may be performed by the server 20.

FIG. 26 is a flowchart illustrating an eyelid redness prediction method.

Referring to FIG. 26 , the server 20 acquires a facial image in stepS400, preprocesses the acquired facial image in step S410, inputs thepreprocessed image to the above-described fourth prediction model(eyelid redness prediction model) in step S420, and acquires an outputvalue of the fourth prediction model in step S430.

Acquisition of Facial Image

The server 20 acquires a facial image in step S400. The server 20 mayacquire the facial image from the user terminal 10.

Preprocessing of Facial Image

The server 20 may preprocess the acquired facial image in step S410. Theserver 20 may perform, on the acquired facial image, iris segmentation,eye outline segmentation, masking, cropping, and resizing, which aredescribed above.

Segmentation Processing

The server 20 may perform the iris segmentation and the eye outlinesegmentation, and the server 20 may thus determine the pixelscorresponding to the iris and the pixels corresponding to the inside ofthe outline of the eye within the acquired facial image. The server 20may determine the coordinate values of the pixels corresponding to theiris and the coordinate values of the pixels corresponding to the insideof the outline of the eye.

However, in performing the eyelid redness prediction method, asdescribed later, iris segmentation needs to be performed when maskingprocessing is performed, but it is allowed not to perform irissegmentation when masking processing is not performed.

Masking Processing

The server 20 may perform the second masking on the facial image on thebasis of information on the determined pixels. Through the secondmasking processing, the server 20 may remove the pixel values of thepixels corresponding to the iris (cornea) exposed to the outside amongthe pixels included in the facial image. Accordingly, the pixel valuesof the pixels corresponding to the region excluding the iris (cornea) ofthe left eye and the iris (cornea) of the right eye may remain to be theoriginal pixel values, but the pixel values of the pixels correspondingto the iris (or cornea) of the left eye and the iris (or cornea) of theright eye may be removed or changed to other values.

However, in performing the eyelid redness prediction method, performingpreprocessing of masking an iris (cornea) is advantageous in severalaspects, but it is allowed not to perform masking on an iris.

Cropping Processing

The server 20 may crop the masked facial image. The server 20 may cropthe masked facial image to generate a left eye cropped image and a righteye cropped image. In performing the eyelid redness prediction method,the server 20 may use the third cropping (eyelid-included cropping)method among the cropping methods described above. Since the thirdcropping method has already been described in detail, a detaileddescription thereof will be omitted here.

Resizing Processing and Lateral Inversion Processing

The server 20 may resize the size of the left eye cropped image and sizeof the right eye cropped image to a predetermined size.

In the meantime, when the fourth prediction model is not dualized, butrealized as one model, the server 20 may laterally invert either theleft eye cropped image or the right eye cropped image as describedabove. The server 20 does not laterally invert the other among the lefteye cropped image and the right eye cropped image. Herein, it isdetermined that the criterion for determining which one of the left eyeimage and the right eye image is subjected to lateral inversion is thesame as the criterion applied when the fourth prediction model istrained. That is, in training the fourth prediction model, when the lefteye image is inverted and the right eye image is not inverted, theserver 20 inverts the left eye image and does not invert the right eyeimage, similarly.

As described above, in realizing the fourth prediction model, when thefourth prediction model is dualized to the fourth left eye predictionmodel and the fourth right eye prediction model, the server 20 may notperform lateral inversion processing.

In the meantime, it has been described that when preprocessing isperformed, segmentation, masking processing, cropping processing,resizing processing, and lateral inversion processing are performed, butthe sequence of these types of preprocessing may be changed within arange capable of achieving the purpose of the eyelid redness predictionmethod disclosed in the present application.

Input of Preprocessed Image

The server 20 may input the preprocessed image to the fourth predictionmodel in step S420.

When the fourth prediction model is not dualized and is realized as onemodel, the server 20 inputs the right eye preprocessed image and thelaterally inverted left eye preprocessed image to the fourth predictionmodel in order.

In realizing the fourth prediction model, when the fourth predictionmodel is dualized to the fourth left eye prediction model and the fourthright eye prediction model, the server 20 inputs the left eyepreprocessed image to the fourth left eye prediction model and inputsthe right eye preprocessed image to the fourth right eye predictionmodel. Alternatively, the server 20 may input the left eye preprocessedimage to the fourth left eye prediction model, may input the laterallyinverted left eye preprocessed image to the fourth right eye predictionmodel, may input the right eye preprocessed image to the fourth righteye prediction model, and may input the laterally inverted right eyepreprocessed image to the fourth left eye prediction model.

In realizing the fourth prediction model, when the fourth predictionmodel is not dualized and is realized as one model and, simultaneously,is trained to be capable of determining whether there is redness ofeyelid without distinguishing between a left eye image and a right eyeimage, the server 20 may input the left eye preprocessed image and theright eye preprocessed image to the fourth prediction model withoutlateral inversion. Alternatively, the server 20 may input the left eyepreprocessed image, the laterally inverted left eye preprocessed image,the right eye preprocessed image, and the laterally inverted righteyepreprocessed image to the fourth prediction model.

Eyelid Redness Prediction Result

The server 20 may output a result value output from the fourthprediction model in step S430. The result value may be a probabilityvalue that is predicted with respect to eyelid redness captured in animage. On the basis of a predetermined threshold value, when thepredicted probability value is equal to or greater than the thresholdvalue, the server 20 determines that there is eyelid redness, or whenthe predicted probability value is less than the threshold value, theserver 20 determines that there is no eyelid redness.

The server 20 may acquire both a prediction result for the left eye anda prediction result for the right eye.

When the server 20 inputs the left eye preprocessed image to the fourthleft eye prediction model, inputs the laterally inverted left eyepreprocessed image to the fourth right eye prediction model, inputs theright eye preprocessed image to the fourth right eye prediction model,and inputs the laterally inverted right eye preprocessed image to thefourth left eye prediction model, the server 20 may obtain a predictionresult for the left eye considering both a result obtained by inputtingthe left eye preprocessed image to the fourth left eye prediction modeland a result obtained by inputting the laterally inverted left eyepreprocessed image to the fourth right eye prediction model. Herein, theserver 20 may obtain a prediction result for the right eye consideringboth a result obtained by inputting the right eye preprocessed image tothe fourth right eye prediction model and a result obtained by inputtingthe laterally inverted right eye preprocessed image to the fourth lefteye prediction model.

For example, the server 20 may obtain a prediction result for the lefteye on the basis of whether an average value of the result obtained byinputting the left eye preprocessed image to the fourth left eyeprediction model and the result obtained by inputting the laterallyinverted left eye preprocessed image to the fourth right eye predictionmodel is equal to or greater than the threshold value.

As another example, when a value of either the result obtained byinputting the left eye preprocessed image to the fourth left eyeprediction model or the result obtained by inputting the laterallyinverted left eye preprocessed image to the fourth right eye predictionmodel is equal to or greater than the above-described threshold value,the server 20 may predict that the eyelid redness is present in the lefteye.

As still another example, when both the result obtained by inputting theleft eye preprocessed image to the fourth left eye prediction model andthe result obtained by inputting the laterally inverted left eyepreprocessed image to the fourth right eye prediction model are equal toor greater than the above-described threshold value, the server 20 maypredict that the conjunctiva of the left eye is hyperemic.

When the server 20 inputs the left eye preprocessed image, the laterallyinverted left eye preprocessed image, the right eye preprocessed image,and the laterally inverted right eye preprocessed image to the fourthprediction model that is not dualized, the server 20 may obtain aprediction result for the left eye considering both a result obtained byinputting the left eye preprocessed image to the fourth prediction modeland a result obtained by inputting the laterally inverted left eyepreprocessed image to the fourth prediction model. Herein, the server 20may obtain a prediction result for the right eye considering both aresult obtained by inputting the right eye preprocessed image to thefourth prediction model and a result obtained by inputting the laterallyinverted right eye preprocessed image to the fourth prediction model.

For example, the server 20 may obtain a prediction result for the lefteye on the basis of whether an average value of the result obtained byinputting the left eye preprocessed image to the fourth prediction modeland the result obtained by inputting the laterally inverted left eyepreprocessed image to the fourth prediction model is equal to or greaterthan the threshold value.

As another example, when a value of either the result obtained byinputting the left eye preprocessed image to the fourth prediction modelor the result obtained by inputting the laterally inverted left eyepreprocessed image to the fourth prediction model is equal to or greaterthan the above-described threshold value, the server 20 may predict thatthere is eyelid redness of the left eye.

As still another example, when both the result obtained by inputting theleft eye preprocessed image to the fourth prediction model and theresult obtained by inputting the laterally inverted left eyepreprocessed image to the fourth prediction model are equal to orgreater than the above-described threshold value, the server 20 maypredict that there is eyelid redness of the left eye.

The above-described method may be similarly applied in determiningwhether there is eyelid redness of the right eye.

9. Eyelid Edema Prediction Method

The eyelid edema prediction method described in the present applicationmay be performed by the server 20.

FIG. 27 is a flowchart illustrating an eyelid edema prediction method.

Referring to FIG. 27 , the server 20 acquires a facial image in stepS500, preprocesses the acquired facial image in step S510, inputs thepreprocessed image to the above-described fifth prediction model (eyelidedema prediction model) in step S520, and acquires an output value ofthe fifth prediction model in step S530.

The eyelid edema prediction method is the same as or very similar to theeyelid redness prediction method except that the fifth prediction modelis used instead of the fourth prediction model and a finally acquiredresult value is a predicted value for whether there is eyelid edema, soa detailed description of the eyelid edema prediction method will beomitted.

As described above, the eyelid redness prediction method and the eyelidedema prediction method use the same image preprocessing methods, butonly the prediction models to which the preprocessed images are inputare different from each other. Therefore, after image preprocessing asdescribed above, the images may be input to different prediction models.

10. Method of Predicting Clinical Activity Score for Thyroid Eye Disease

Hereinafter, described will be a method, which is described in thepresent application, of predicting a clinical activity score for thyroideye disease.

FIG. 28 is a diagram illustrating a method of predicting a clinicalactivity score for thyroid eye disease.

The server 20 may acquire a facial image.

The server 20 performs two different types of preprocessing on onefacial image. First preprocessing (hereinafter, first preprocessing)includes iris segmentation, eye outline segmentation, first masking,second cropping (eye outline cropping), resizing, and lateral inversion,and second preprocessing (hereinafter, second preprocessing) includesiris segmentation, eye outline segmentation, second masking, thirdcropping (eyelid-included cropping), resizing, and lateral inversion.However, as described in the eyelid redness prediction method, irissegmentation and second masking may be omitted.

The server 20 acquires a first preprocessed image by performing firstpreprocessing on the acquired facial image, and the first preprocessedimage includes a first left eye preprocessed image and a first right eyepreprocessed image. Herein, either the first left eye preprocessed imageor the first right eye preprocessed image is an image on which lateralinversion is processed. Furthermore, as already described in detail,since the first preprocessed image is an image obtained using secondcropping, the number of pixels corresponding to the eyelids within thefirst preprocessed image is minimized and the pixels corresponding tothe conjunctivas exposed to the outside and the lacrimal caruncles areincluded. Furthermore, the first preprocessed image is an image obtainedusing first masking, the pixel values of pixels corresponding to irises(or corneas) and eyelids (upper eyelid and lower eyelid) are removed,but the pixel values of pixels corresponding to the conjunctivas exposedto the outside and the lacrimal caruncles remain.

In addition, the server 20 acquires a second preprocessed image byperforming second preprocessing on the acquired facial image, and thesecond preprocessed image includes a second left eye preprocessed imageand a second right eye preprocessed image. Herein, either the secondleft eye preprocessed image or the second right eye preprocessed imageis an image on which lateral inversion is performed. Furthermore, asalready described in detail, since the second preprocessed image is animage obtained by using third cropping, the second preprocessed imageincludes sufficient pixels corresponding to eyelids. Furthermore, whenthe second masking method is used to obtain the second preprocessedimage, the pixel values of the pixels corresponding to the irises (orcorneas) and eyelids (upper eyelid and lower eyelid) may be removed.

The server 20 inputs the first preprocessed image (the first left eyepreprocessed image and the first right eye preprocessed image) to thefirst prediction model in order. The server 20 obtains a result value(probability value) of the first prediction model for the first left eyepreprocessed image, and determines, on the basis of the result value,whether there is conjunctival hyperemia of the left eye. In addition,the server 20 obtains a result value (probability value) of the firstprediction model for the first right eye preprocessed image, anddetermines, on the basis of the result value, whether there isconjunctival hyperemia of the right eye.

The server 20 synthesizes a determination result for the left eye and adetermination result for the right eye, and finally determines whetherthere is conjunctival hyperemia of both eyes. For example, whendetermining that there is conjunctival hyperemia of either the left eyeor the right eye or both, the server 20 finally determines that there isconjunctival hyperemia.

Next, the server 20 inputs the first preprocessed image (the first lefteye preprocessed image and the first right eye preprocessed image) tothe second prediction model in order. The server 20 obtains a resultvalue (probability value) of the second prediction model for the firstleft eye preprocessed image, and determines, on the basis of the resultvalue, whether there is conjunctival edema of the left eye. In addition,the server 20 obtains a result value (probability value) of the secondprediction model for the first right eye preprocessed image, anddetermines, on the basis of the result value, whether there isconjunctival edema of the right eye.

The server 20 synthesizes a determination result for the left eye and adetermination result for the right eye, and finally determines whetherthere is conjunctival edema of both eyes. For example, when determiningthat there is conjunctival edema of either the left eye or the right eyeor both, the server 20 finally determines that there is conjunctivaledema.

Next, the server 20 inputs the first preprocessed image (the first lefteye preprocessed image and the first right eye preprocessed image) tothe third prediction model in order. The server 20 obtains a resultvalue (probability value) of the third prediction model for the firstleft eye preprocessed image, and determines, on the basis of the resultvalue, whether there is lacrimal edema of the left eye. In addition, theserver 20 obtains a result value (probability value) of the thirdprediction model for the first right eye preprocessed image, anddetermines, on the basis of the result value, whether there is lacrimaledema of the right eye.

The server 20 synthesizes a determination result for the left eye and adetermination result for the right eye, and finally determines whetherthere is lacrimal edema of both eyes. For example, when determining thatthere is lacrimal edema of either the left eye or the right eye or both,the server 20 finally determines that there is lacrimal edema.

The server 20 inputs the second preprocessed image (the second left eyepreprocessed image and the second right eye preprocessed image) to thefourth prediction model in order. The server 20 obtains a result value(probability value) of the fourth prediction model for the second lefteye preprocessed image, and determines, on the basis of the resultvalue, whether there is eyelid redness of the left eye. In addition, theserver 20 obtains a result value (probability value) of the fourthprediction model for the second right eye preprocessed image, anddetermines, on the basis of the result value, whether there is eyelidredness of the right eye.

The server 20 synthesizes a determination result for the left eye and adetermination result for the right eye, and finally determines whetherthere is eyelid redness of both eyes. For example, when determining thatthere is eyelid redness of either the left eye or the right eye or both,the server 20 finally determines that there is eyelid redness.

The server 20 inputs the second preprocessed image (the second left eyepreprocessed image and the second right eye preprocessed image) to thefifth prediction model in order. The server 20 obtains a result value(probability value) of the fifth prediction model for the second lefteye preprocessed image, and determines, on the basis of the resultvalue, whether there is eyelid edema of the left eye. In addition, theserver 20 obtains a result value (probability value) of the fifthprediction model for the second right eye preprocessed image, anddetermines, on the basis of the result value, whether there is eyelidedema of the right eye.

The server 20 synthesizes a determination result for the left eye and adetermination result for the right eye, and finally determines whetherthere is eyelid edema of both eyes. For example, when determining thatthere is eyelid edema of either the left eye or the right eye or both,the server 20 finally determines that there is eyelid edema.

When it is determined that there is a symptom through a predictionmodel, the server 20 may give a predetermined score (for example, ascore of 1) for the symptom. The server may give scores for fiverespective symptoms according to determination results for the fiveprediction models, and may also obtain a value obtained by adding allthe scores.

It has been described that the above-described method, which isdescribed in the present application, of predicting a clinical activityscore for thyroid eye disease is performed by the server 20.

However, the above-described method may be performed by a user terminal10. Alternatively, preprocessing of the above-described methods may beperformed by the user terminal 10, and determination for each of thesymptoms may be performed by the server. That is, the above-describedsteps may be appropriately distributed to the user terminal 10 and theserver 20 and performed.

11. Hospital Visit Recommendation Method Based on Continuous Monitoringof Clinical Activity Score for Thyroid Eye Disease

Hereinafter, described will be a method of continuously monitoring aclinical activity score for thyroid eye disease, and a method ofrecommending a hospital visit on the basis of the monitoring methoddescribed in the present application.

FIG. 29 is a diagram illustrating a method of continuously monitoring aclinical activity score for thyroid eye disease, and a method ofrecommending a hospital visit on the basis of the monitoring methoddescribed in the present application.

A user terminal 10 may output a guide for acquiring a facial image,through a display 112 in step S600.

The user terminal 10 may output, through the display 112, an image (forexample, an image of a user's face) captured in real time through acamera 140. Herein, the guide may be output together.

The user terminal 10 may acquire a facial image of the user's facethrough the camera 140 in step S610.

The user terminal 10 may transmit the acquired facial image to theserver 20 in step S620.

The user terminal 10 may output, through the display, a graphical userinterface (GUI) for receiving a user input with respect to spontaneousretrobulbar pain and pain on an attempted upward or downward gaze amonga total of seven items considered in determining a clinical activityscore for thyroid eye disease. Next, the user terminal 10 may receivethe user's response to the two items in step S630. The user terminal 10may give, on the basis of the input response of the user, apredetermined score (for example, a score of 1) for each of the items.For example, when the user provides an input that the user hasspontaneous retrobulbar pain, the user terminal 10 may give a score of 1for the item. In addition, when the user provides an input that the userhas pain on an attempted upward or downward gaze, the user terminal 10may give a score of 1 for the item.

The user terminal 10 may receive, from the server 20 in step S640,determination results or a total score thereof for redness of aconjunctiva, swelling of a conjunctiva, swelling of a lacrimal caruncle,redness of an eyelid, and swelling of an eyelid among the total of sevenitems considered in determining a clinical activity score for thyroideye disease on the basis of the acquired facial image.

The user terminal 10 may calculate a final clinical activity score forthyroid eye disease in step S650 on the basis of the scores determinedby the user input and the scores received from the server 20 or thescores determined on the basis of the determination results receivedfrom the server.

The user terminal 10 may store the time at which the facial image of theuser is acquired, or the time at which a calculation value of the finalclinical activity score for thyroid eye disease is acquired, or thecorresponding time (hereinafter, referred to as measurement time,yy/mm/dd, hh:mm) in a memory 130 together with the calculated clinicalactivity score. Alternatively, the user terminal 10 may transmit theabove-described measurement time and a clinical activity scorecorresponding thereto to the server 20. Herein, the server 20 may storethe measurement time and the clinical activity score in association withthe user terminal 10 or the user in step S660.

In the meantime, the measurement time includes information on the date.The measurement time may include both information on the date andinformation on the hour and/or minute. Alternatively, the measurementtime may include only the information on the date, and may not includethe information on the hour or minute.

The user terminal 10 may output, on the basis of the calculated clinicalactivity score, information for recommending that the user visit thehospital and have a detailed medical examination, through the display112 in step S670.

When the calculated clinical activity score is less than a score of 3,the user terminal 10 may output information indicating that there is norisk of thyroid eye disease, through the display 112.

When the calculated clinical activity score is a score of 3 or 4, theuser terminal 10 may output information indicating that there is no riskof thyroid eye disease, through the display 112, or may outputinformation for recommending that the user visit the hospital and have adetailed medical examination, through the display 112, by choice.

When the calculated clinical activity score is equal to or greater thana score of 5, the user terminal 10 may output information forrecommending that the user visit the hospital and have a detailedmedical examination, through the display 112.

When the calculated clinical activity score is a score of 3 or 4, theclinical activity score that was measured a predetermined period of time(for example, one week) before the corresponding time point isdetermined, and it may be determined whether the clinical activity scorewas a score of 3 or 4 during the corresponding time interval(hereinafter, the monitoring time interval). Herein, when a score of 3or 4 occurred one or more times during the monitoring time interval, theuser terminal 10 outputs information for recommending that the uservisit the hospital and have a detailed medical examination, through thedisplay 112. When a score of 3 or 4 never occurred during the monitoringtime interval, the user terminal 10 outputs information indicating thatthere is no risk of thyroid eye disease, through the display 112.

When the calculated clinical activity score is equal to or greater thana score of 3, the user terminal 10 may output information forrecommending that the user visit the hospital and have a detailedmedical examination, through the display 112 without additionaldetermination for past records.

According to the above description, in outputting information to theuser through the user terminal 10, outputting the information visuallythrough the display 112 has been described as an example, but in somecases, the information may be audibly output through a speaker.

In addition, it has been described that the method of continuouslymonitoring a clinical activity score for thyroid eye disease, and themethod of recommending a hospital visit on the basis of the monitoringmethod described in the present application are performed by the userterminal 10. However, the steps of the above-described methods may beappropriately distributed to the user terminal 10 and the server 20 andperformed. For example, when the measurement time and the clinicalactivity score are transmitted to the server 20 and stored, whether ascore of 3 or 4 occurred during the monitoring time interval may bedetermined by the server 20.

12. Experimental Example #1

(1) Preparation of Facial Images

1,020 facial images were prepared. Each of the facial images was animage including both a left eye and a right eye, and was an imageobtained according to a predetermined photographing structure.

(2) Securing Labeling Information of Facial Images

For each of the 1,020 facial images, information on conjunctivalhyperemia, conjunctival edema, lacrimal edema, eyelid redness, andeyelid edema for the left eye, and information on conjunctivalhyperemia, conjunctival edema, lacrimal edema, eyelid redness, andeyelid edema for the right eye were secured, and the data were used aslabeling data.

Among 1,020 data sets, 714 data sets were used as a training data set(training set), 102 data sets were used as a validation sets, and 204data sets were used as a test set.

Furthermore, dividing the 1,020 data sets into a training data set, avalidation set, and a test set was randomly performed 30 times, and afirst training data set group to a 30th training data set group werecreated accordingly.

(3) Securing first preprocessed images and second preprocessed images offacial images

For each of the 1,020 facial images, second cropping processing (eyeoutline cropping) was performed on each of the left eye and the righteye in the above-described manner to secure a first left eyepreprocessed image and a first right eye preprocessed image. Herein, alaterally inverted image was used as the first right eye preprocessedimage, and a non-laterally inverted image was used as the first left eyepreprocessed image. In the meantime, both the first left eyepreprocessed image and the first right eye preprocessed image wereimages on which the above-described first masking processing wasperformed.

For each of the 1,020 facial images, third cropping processing(eyelid-included cropping) was performed on each of the left eye and theright eye in the above-described manner to secure a second left eyepreprocessed image and a second right eye preprocessed image. Herein, alaterally inverted image was used as the second right eye preprocessedimage, and a non-laterally inverted image was used as the second lefteye preprocessed image. In the meantime, both the second left eyepreprocessed image and the second right eye preprocessed image wereimages on which masking processing was not performed.

(4) Training of First to Fifth Prediction Models According toExperimental Example #1

The secured first preprocessed images and the secured pieces of labelinginformation thereof, and the secured second preprocessed images and thesecured pieces of labeling information thereof were used in training thefirst to fifth prediction models.

As the prediction models, the models using the above-described ViT asthe backbone architecture were used, and each of the prediction modelswas trained with unification as one model without dividing into a lefteye prediction model and a right eye prediction model.

(5) Acquisition of Prediction Result for Each Symptom by UsingPrediction Models

Prediction results were acquired using the test data sets for thetrained first to fifth prediction models. Herein, a laterally invertedpreprocessed image was used as a right eye image, and a non-laterallyinverted preprocessed image was as a left eye image.

(6) Accuracy, Sensitivity, Specificity, Positive Predictive Value (PPV),and Negative Predictive Value (NPV) of Eyelid Redness Prediction ModelAccording to Experimental Example #1

The values shown in [Table 1] are average values of accuracy,sensitivity, specificity, PPV, and NPV measured for the first to fifthprediction models that were trained for each of the 30 data set groupsaccording to the above-described experimental example #1.

TABLE 1 Accuracy Sensitivity Specificity PPV NPV (%) (%) (%) (%) (%)Conjunctival 80.80 86.84 76.40 73.58 89.10 hyperemia (first predictionmodel) Conjunctival edema 89.12 42.18 94.82 50.39 93.15 (secondprediction model) Lacrimal edema 88.14 55.54 92.52 53.14 93.96 (thirdprediction model) Eyelid redness 72.42 73.94 71.55 59.10 84.17 (fourthprediction model) Eyelid edema 81.29 86.38 53.52 91.16 43.59 (fifthprediction model)

13. Experimental Example #2

(1) Preparation of Facial Images

The facial images used in experimental example #1 were used as theywere.

(2) Securing Labeling Information of Facial Images

The pieces of labeling information of the facial images used inexperimental example #1 were used as they were.

(3) Securing First Preprocessed Images and Second Preprocessed Images ofFacial Images

For each of the 1,020 facial images, second cropping processing (eyeoutline cropping) is performed on each of the left eye and the right eyein the above-described manner to secure first preprocessed images.Unlike experimental example #1, a first left eye preprocessed image notsubjected to lateral inversion, a first left eye preprocessed imagesubjected to lateral inversion, a first right eye preprocessed image notsubjected to lateral inversion, and a first right eye preprocessed imagesubjected to lateral inversion were secured, and were used in training.Herein, both the first left eye preprocessed image and the first righteye preprocessed image were images on which the above-described firstmasking processing was performed.

For each of the 1,020 facial images, third cropping processing(eyelid-included cropping) was performed on each of the left eye and theright eye in the above-described manner to secure second preprocessedimages. Unlike experimental example #1, a second left eye preprocessedimage not subjected to lateral inversion, a second left eye preprocessedimage subjected to lateral inversion, a second right eye preprocessedimage not subjected to lateral inversion, and a second right eyepreprocessed image subjected to lateral inversion were secured, and wereused in training. Herein, both the second left eye preprocessed imageand the second right eye preprocessed image were images on which maskingprocessing was not performed.

(4) Training of First to Fifth Prediction Models According toExperimental Example #2

The secured first preprocessed images and the secured pieces of labelinginformation thereof, and the secured second preprocessed images and thesecured pieces of labeling information thereof were used in training thefirst to fifth prediction models.

As the prediction models, the models using the above-described ViT asthe backbone architecture were used, and each of the prediction modelswas trained being dualized into a left eye prediction model and a righteye prediction model. In particular, when the left eye prediction modelswere trained, left eye preprocessed images not subjected to lateralinversion and right eye preprocessed images subjected to lateralinversion were used, and when the right eye prediction models weretrained, right eye preprocessed images not subjected to lateralinversion and left eye preprocessed images subjected to lateralinversion were used.

(5) Acquisition of Prediction Result for Each Symptom by UsingPrediction Models

Prediction results were acquired using the test data sets for thetrained first to fifth prediction models. Herein, the prediction resultsfor the right eye were acquired by inputting a right eye preprocessedimage not subjected to lateral inversion to each of the right eyeprediction models, and the prediction results for the left eye wereacquired by inputting a left eye preprocessed image not subjected tolateral inversion to each of the left eye prediction models.

(6) Accuracy, Sensitivity, Specificity, Positive Predictive Value (PPV),and Negative Predictive Value (NPV) of Eyelid Redness Prediction ModelAccording to Experimental Example #2

The values shown in [Table 2] are average values of accuracy,sensitivity, specificity, PPV, and NPV measured for the first to fifthprediction models that were trained for each of the 30 data set groupsaccording to the above-described experimental example #2.

TABLE 2 Accuracy Sensitivity Specificity PPV NPV (%) (%) (%) (%) (%)Conjunctival 79.54 87.26 73.99 71.84 89.19 hyperemia (first predictionmodel) Conjunctival edema 89.22 45.04 94.61 52.53 93.48 (secondprediction model) Lacrimal edema 88.35 49.51 93.70 55.48 93.20 (thirdprediction model) Eyelid redness 76.13 70.65 79.08 64.63 83.86 (fourthprediction model) Eyelid edema 81.47 86.51 54.44 91.22 44.62 (fifthprediction model)

-   -   1: System    -   10: User terminal    -   20: Server

1. (canceled)
 2. (canceled)
 3. (canceled)
 4. (canceled)
 5. (canceled) 6.(canceled)
 7. (canceled)
 8. (canceled)
 9. (canceled)
 10. (canceled) 11.A computer-implemented method of managing a thyroid eye disease,comprising: outputting a first screen for receiving an evaluation of auser's subjective pain through a user terminal; based on a prestoredscore determination algorithm and a first user input, calculating ascore of a spontaneous retrobulbar pain and a score of a pain onattempted upward or downward gaze, wherein the first user inputcomprises an input obtained as a response to the first screen;outputting a second screen for receiving a facial image, wherein, at thesecond screen, a photograph guide appears on the user terminal such thatat least one eye of the user appears on the facial image; based on oneor more sign prediction models and the facial image, obtaining a scoreof a conjunctival hyperemia, a score of a conjunctival edema, a score ofa lacrimal edema, a score of an eyelid redness and a score of an eyelidedema, wherein the one or more sign prediction models is pretrained witha training data set that comprises clinical images captured on ahospital and a doctor's judgments on the clinical images; calculating afinal score by adding the scores of the spontaneous retrobulbar pain,the pain on attempted upward or downward gaze, the conjunctivalhyperemia, the conjunctival edema, the lacrimal edema, the eyelidredness, and the eyelid edema with a same weight; and determiningwhether the user is recommended to visit a hospital by comparing thefinal score and final criteria.
 12. The method of claim 11, whereindetermining whether the user is recommended to visit comprisesdetermining to recommend visiting the hospital when the final score is 3or more.
 13. The method of claim 11, wherein determining whether theuser is recommended to visit comprises: determining to recommendvisiting the hospital when the final score is 5 or more, and determiningto recommend visiting the hospital when the final score is 3 or 4 and atleast one of final scores of a predetermined period is 3 or more. 14.The method of claim 13, wherein the predetermined period is set as oneweek shortly before a day that the final score is calculated.
 15. Themethod of claim 11, wherein the score of the conjunctival hyperemia iscalculated based on the facial image, a first prediction model and afirst evaluation criteria, wherein the score of the conjunctival edemais calculated based on the facial image, a second prediction model and asecond evaluation criteria, wherein the score of the lacrimal edema iscalculated based on the facial image, a third prediction model and athird evaluation criteria, wherein the score of the eyelid redness iscalculated based on the facial image, a fourth prediction model and afourth evaluation criteria, and wherein the score of the eyelid edema iscalculated based on the facial image, a fifth prediction model and afifth evaluation criteria.
 16. The method of claim 15, wherein the firstevaluation criteria is satisfied when a probability value of the firstprediction model is greater than a first threshold, wherein the secondevaluation criteria is satisfied when a probability value of the secondprediction model is greater than a second threshold, wherein the thirdevaluation criteria is satisfied when a probability value of the thirdprediction model is greater than a third threshold, wherein the fourthevaluation criteria is satisfied when a probability value of the fourthprediction model is greater than a fourth threshold, and wherein thefifth evaluation criteria is satisfied when a probability value of thefifth prediction model is greater than a fifth threshold.
 17. The methodof claim 16, wherein at least one of the first threshold, the secondthreshold, the third threshold, the fourth threshold, and the fifththreshold is different from another one of the first threshold, thesecond threshold, the third threshold, the fourth threshold, and thefifth threshold.
 18. The method of claim 16, wherein the score of theconjunctival hyperemia is determined as 1 when: 1) a probability valueof the first prediction model by inputting a left eye image of thefacial image is greater than the first threshold, and/or 2) aprobability value of the first prediction model by inputting a right eyeimage of the facial image is greater than the first threshold, whereinthe score of the conjunctival edema is determined as 1 when: 1) aprobability value of the second prediction model by inputting a left eyeimage of the facial image is greater than the second threshold, and/or2) a probability value of the second prediction model by inputting aright eye image of the facial image is greater than the secondthreshold, wherein the score of the lacrimal edema is determined as 1when: 1) a probability value of the third prediction model by inputtinga left eye image of the facial image is greater than the thirdthreshold, and/or 2) a probability value of the third prediction modelby inputting a right eye image of the facial image is greater than thethird threshold, wherein the score of the eyelid redness is determinedas 1 when: 1) a probability value of the fourth prediction model byinputting a left eye image of the facial image is greater than thefourth threshold, and/or 2) a probability value of the fourth predictionmodel by inputting a right eye image of the facial image is greater thanthe fourth threshold, and wherein the score of the eyelid edema isdetermined as 1 when: 1) a probability value of the fifth predictionmodel by inputting a left eye image of the facial image is greater thanthe fifth threshold, and/or 2) a probability value of the fifthprediction model by inputting a right eye image of the facial image isgreater than the fifth threshold.
 19. The method of claim 11, wherein,at the first screen, a first question as to whether the spontaneousretrobulbar pain is felt and a second question as to whether the pain onattempted upward or downward gaze is felt appear on the user terminal.20. The method of claim 19, wherein, at the first screen, a firstinterface for receiving a yes regarding the first question, a firstinterface for receiving a no regarding the first question, a secondinterface for receiving a yes regarding the second question, and asecond interface for receiving a no regarding the second question arealso appears on the user terminal.
 21. The method of claim 20, whereinthe score of spontaneous retrobulbar pain is determined as 1 when a userinput corresponding to the first interface is received, wherein thescore of pain on attempted upward or downward gaze is determined as 1when a user input corresponding to a third interface is received.
 22. Acomputing device comprising: a processor; and a memory operativelycoupled to the processor, the memory containing instructions executableby the processor to cause the computing device to perform operationscomprising: outputting a first screen for receiving an evaluation of auser's subjective pain through a user terminal; based on a prestoredscore determination algorithm and a first user input, calculating ascore of a spontaneous retrobulbar pain and a score of a pain onattempted upward or downward gaze, wherein the first user inputcomprises an input obtained as a response to the first screen;outputting a second screen for receiving a facial image, wherein, at thesecond screen, a photograph guide appears on the user terminal such thatat least one eye of a user appears on the facial image; based on one ormore sign prediction models and the facial image, obtaining a score of aconjunctival hyperemia, a score of a conjunctival edema, a score of alacrimal edema, a score of an eyelid redness and a score of an eyelidedema, wherein the one or more sign prediction models is pretrained witha training data set that comprises clinical images captured on ahospital and a doctor's judgments on the clinical images; calculating afinal score by adding the scores of the spontaneous retrobulbar pain,the pain on attempted upward or downward gaze, the conjunctivalhyperemia, the conjunctival edema, the lacrimal edema, the eyelidredness, and the eyelid edema with a same weight; and determiningwhether the user is recommended to visit a hospital by comparing thefinal score and final criteria.
 23. The computing device of claim 22,wherein determining whether the user is recommended to visit comprisesdetermining to recommend visiting the hospital when the final score is 3or more.
 24. The computing device of claim 22, wherein determiningwhether the user is recommended to visit comprises: determining torecommend visiting the hospital when the final score is 5 or more, anddetermining to recommend visiting the hospital when the final score is 3or 4 and at least one of final scores of a predetermined period is 3 ormore.
 25. The computing device of claim 24, wherein the predeterminedperiod is set as one week shortly before a day that the final score iscalculated.
 26. The computing device of claim 25, wherein the score ofthe conjunctival hyperemia is calculated based on the facial image, afirst prediction model and a first evaluation criteria, wherein thescore of the conjunctival edema is calculated based on the facial image,a second prediction model and a second evaluation criteria, wherein thescore of the lacrimal edema is calculated based on the facial image, athird prediction model and a third evaluation criteria, wherein thescore of the eyelid redness is calculated based on the facial image, afourth prediction model and a fourth evaluation criteria, and whereinthe score of the eyelid edema is calculated based on the facial image, afifth prediction model and a fifth evaluation criteria.
 27. Thecomputing device of claim 26, wherein the score of the conjunctivalhyperemia is determined as 1 when: 1) a probability value of the firstprediction model by inputting a left eye image of the facial image isgreater than a first threshold, and/or 2) a probability value of thefirst prediction model by inputting a right eye image of the facialimage is greater than the first threshold, wherein the score of theconjunctival edema is determined as 1 when: 1) a probability value ofthe second prediction model by inputting a left eye image of the facialimage is greater than a second threshold, and/or 2) a probability valueof the second prediction model by inputting a right eye image of thefacial image is greater than the second threshold, wherein the score ofthe lacrimal edema is determined as 1 when: 1) a probability value ofthe third prediction model by inputting a left eye image of the facialimage is greater than a third threshold, and/or 2) a probability valueof the third prediction model by inputting a right eye image of thefacial image is greater than the third threshold, wherein the score ofthe eyelid redness is determined as 1 when: 1) a probability value ofthe fourth prediction model by inputting a left eye image of the facialimage is greater than a fourth threshold, and/or 2) a probability valueof the fourth prediction model by inputting a right eye image of thefacial image is greater than the fourth threshold, and wherein the scoreof the eyelid edema is determined as 1 when: 1) a probability value ofthe fifth prediction model by inputting a left eye image of the facialimage is greater than a fifth threshold, and/or 2) a probability valueof the fifth prediction model by inputting a right eye image of thefacial image is greater than the fifth threshold.
 28. The computingdevice of claim 22, wherein, at the first screen, a first question as towhether the spontaneous retrobulbar pain is felt and a second questionas to whether the pain on attempted upward or downward gaze is feltappears on the user terminal, and a first interface for receiving a yesregarding the first question, a first interface for receiving a noregarding the first question, a second interface for receiving a yesregarding the second question, and a second interface for receiving a noregarding the second question are also appears on the user terminal. 29.The computing device of claim 28, wherein the score of spontaneousretrobulbar pain is determined as 1 when a user input corresponding tothe first interface is received, wherein the score of pain on attemptedupward or downward gaze is determined as 1 when a user inputcorresponding to a third interface is received.
 30. A computer-readablestorage medium having computer-executable instructions stored thereuponwhich, when executed by one or more processors of a computing device,cause the computing device to perform operations comprising: outputtinga first screen for receiving an evaluation of a user's subjective painthrough a user terminal; based on a prestored score determinationalgorithm and a first user input, calculating a score of a spontaneousretrobulbar pain and a score of a pain on attempted upward or downwardgaze, wherein the first user input comprises an input obtained as aresponse to the first screen; outputting a second screen for receiving afacial image, wherein, at the second screen, a photograph guide appearson the user terminal such that at least one eye of a user appears on thefacial image; based on one or more sign prediction models and the facialimage, obtaining a score of a conjunctival hyperemia, a score of aconjunctival edema, a score of a lacrimal edema, a score of an eyelidredness and a score of an eyelid edema, wherein the one or more signprediction models is pretrained with a training data set that comprisesclinical images captured on a hospital and a doctor's judgments on theclinical images; calculating a final score adding the scores of thespontaneous retrobulbar pain, the pain on attempted upward or downwardgaze, the conjunctival hyperemia, the conjunctival edema, the lacrimaledema, the eyelid redness, and the eyelid edema with the same weight;and determining whether the user is recommended to visit a hospital bycomparing the final score and final criteria.